datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
- 2.8k stars
- 174 forks
- 134 open issues
- Language: Python
- License: Apache-2.0
- updated
Install
git clone https://github.com/datachain-ai/datachain && cp datachain/*.md ~/.claude/agents/- Clone the repository and copy the agent .md definitions into ~/.claude/agents (or .claude/agents inside a project).
- Start a new Claude Code session to load the agents.
- Delegate work to them with the Task/Agent tool or by name.
Editor's note
DataChain is a Python library and MCP server that turns unstructured files stored in S3, GCS, and Azure into typed, versioned datasets queryable at warehouse speed, without copying bytes out of storage. Its three main components are a Compute Engine for parallel and distributed Python processing with async I/O and checkpoint recovery, a Dataset DB backed by Pydantic schemas that tracks versions, file pointers, and lineage in a local SQLite store, and a Knowledge Base that generates markdown summaries from datasets enriched by LLM. The Agent Harness connects all three to Claude Code, Cursor, Codex, GitHub Copilot, and Pi via a single install command such as `datachain skill install --target claude`, exposing tools like `read_storage`, `map`, and `save` so agents can build and reuse named dataset versions like `pets_embeddings@1.0.0` across sessions. Data engineers, MLOps practitioners, and teams running multimodal pipelines benefit most, particularly when agents need persistent data context rather than recomputing from raw files on every run.
3 items in this repository
Use ONLY for abstract DataChain SDK questions — API usage, method signatures, or code patterns — when no specific dataset or bucket is referenced. If the request mentions creating, saving, listing, exploring datasets or buckets, use datachain-knowledge instead.
Use when asked about Studio job analytics — compute hours, user spend, failure rates, cost estimation, cluster usage. Generates and maintains dc-knowledge/jobs/index.md.
Use whenever datasets, cloud storage buckets, or data pipelines are mentioned — creating, saving, querying, listing, exploring, deleting, or processing data in S3, GCS, Azure Blob, or local storage. Also use when running any script that may create datasets as a side effect. Maintains a knowledge base at dc-knowledge/ (JSON + markdown). ALWAYS use this skill when the user creates a dataset, saves pipeline output, runs a data script, or references any storage bucket.
Subagents overview
What people ask about datachain
What is datachain-ai/datachain?
datachain-ai/datachain is subagents for the Claude AI ecosystem. The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure It has 2.8k GitHub stars and its last recorded update is dated 2026-10-23.
How do I install datachain?
You can install datachain by cloning the repository (https://github.com/datachain-ai/datachain) or following the README instructions on GitHub. ClaudeWave also provides quick install blocks on this page.
Is datachain-ai/datachain safe to use?
Our security agent has analyzed datachain-ai/datachain and assigned a Trust Score of 100/100 (tier: Verified). See the full breakdown of passed checks and flags on this page.
Who maintains datachain-ai/datachain?
datachain-ai/datachain is maintained by datachain-ai. The last recorded GitHub activity is dated 2026-10-23, with 134 open issues.
Are there alternatives to datachain?
Yes. On ClaudeWave you can browse similar subagents at /categories/agents, sorted by popularity or recent activity.