data-engineering
Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Use this to design or debug a pipeline, decide batch versus streaming, add data quality checks, handle late or duplicate data, or work out why a dashboard's numbers changed without anyone changing the dashboard.
git clone --depth 1 https://github.com/cbrock84/headcount /tmp/data-engineering && cp -r /tmp/data-engineering/plugins/data-analytics/skills/data-engineering ~/.claude/skills/data-engineeringSKILL.md
# Data engineering Pipelines are production systems whose failures are quiet. A broken service pages someone; a broken pipeline produces plausible numbers that people act on for a week. This is movement and transformation. Schema and semantics belong to `data-analytics:data-modeling`, policy and stewardship to `data-analytics:data-governance`. ## Land raw, transform downstream Keep an immutable copy of source data exactly as received. Transformation logic will be wrong at some point, and raw data is what lets you reprocess rather than re-request from a source that may no longer have it. Business logic belongs downstream where it is visible and testable, not buried in ingestion. The exception is transformation required for privacy — minimization, pseudonymization, dropping fields you have no basis to hold — which belongs at ingest precisely because raw storage is what the obligation attaches to. See `legal-risk:privacy-and-data-protection`. ## Idempotence is the property that matters Every pipeline will be re-run: after a failure, after a fix, after a late-arriving correction. A re-run that double-counts is worse than a failure, because it produces a wrong answer silently. Design for exactly-once effect at the destination — deterministic keys, merges rather than blind appends, partitioned overwrites. Then re-running is safe and recovery stops being frightening. ## Late, duplicate and out-of-order data Real sources deliver all three. Decide explicitly, per pipeline: how late is an event still accepted, what happens to one arriving after its window closed, and how duplicates are identified. Distinguish **event time** from **processing time** and partition on event time. Aggregations built on arrival time silently reassign yesterday's activity to today whenever a delivery is delayed. ## Test data, not just code Unit tests on transformation logic catch the wrong class of failure. Most damage comes from data that is valid but wrong. Assert on the data itself, in the pipeline, and fail loudly: - Row counts within an expected range, not merely non-zero. - Uniqueness of keys, and referential integrity across joins. - Freshness — the newest record is recent enough to be meaningful. - Distribution shifts in important columns. A silent failure is worse than a loud one. Prefer stopping the pipeline to publishing data you do not trust. ## Tooling Warehouses and lakehouses: Snowflake, BigQuery, Databricks, Redshift, and Postgres or DuckDB at small scale, and similar. Ingestion: Fivetran, Airbyte, Stitch, and similar. Transformation: dbt, SQLMesh. Orchestration: Airflow, Dagster, Prefect, and similar. Buy ingestion and build transformation. Connector maintenance returns nothing for the time your team puts into it. ## Never - Transform on ingest for business reasons and discard the raw copy. - Build a pipeline whose re-run double-counts. - Aggregate on processing time when event time is available. - Let a pipeline fail silently and publish stale data as current.
Concentrates marketing and sales effort on a named set of accounts rather than on volume — qualifying whether the model fits your economics at all, building the account list and the buying group inside each, tiering effort against account value, coordinating so the account experiences one campaign rather than several, and measuring account progression instead of leads. Use this to decide whether to run an account-based program, build one, or work out why an existing one produces activity and no pipeline.
Designs orchestrator-and-subagent hierarchies for a repository — splitting agents by exclusive write surface, pairing every producer with an independent auditor, and enforcing the split with a script that runs in CI. Use this whenever the user wants to set up, expand, audit, or fix a multi-agent or subagent structure for a codebase; asks how to divide work between agents; wants agent charters, roles, or a surface map written; or is hitting agents that collide on the same files, review their own work, or drift from their remit. Also use when sizing a roster or deciding whether a new agent is justified.
Governs models and AI systems in production — intended use, evaluation, monitoring, human oversight, documentation, and the decision to deploy or retire. Use this before deploying a model or AI feature, when defining evaluation criteria, when a model's behavior has drifted, when assessing AI risk or regulatory exposure, or when deciding whether an AI system is fit for a consequential decision.
Produces executive-level research — market sizing, competitor mapping, trend analysis, and strategic intelligence — grounded in cited sources with the confidence in each claim made explicit. Use this to analyze a market or industry, map competitors, evaluate a market-entry or build-versus-buy decision, produce a research brief, or assemble evidence for a decision. Also use when comparing options that need a structured, evidence-based verdict rather than an opinion.
Optimizes for AI assistants and AI-generated answers — being retrievable, being cited, and being represented accurately when a model answers on your behalf. Use this when traffic is shifting from links to AI answers, when a brand is misrepresented or absent in AI responses, when planning content for retrieval rather than ranking, or when deciding how AI search changes an existing SEO program.
Improves visibility and conversion in the App Store and Google Play — metadata, keywords, screenshots, ratings, and the listing experience that turns an impression into an install. Use this to audit or optimize an app listing, plan a launch listing, diagnose poor install conversion, or improve store search visibility.
Protects and restores data — backup coverage and scope, retention, immutability against ransomware, and proving restores actually work. Use this to design a backup regime, verify restores, plan retention, protect backups from ransomware, or recover from data loss.
Runs the planning cycle — annual budget, rolling forecast, consolidation of business unit inputs, and the variance analysis that explains actuals against plan. Use this to build or challenge a budget, run a forecast cycle, consolidate departmental submissions, explain why results diverged from plan, or improve forecast accuracy.