Data Pipeline Architect

What it does

Data Pipeline Architect takes a plain-English description of your data flow ("pull customer events from Kafka, join with user records from Postgres, compute 7-day rolling averages, write to BigQuery partitioned by date") and produces the full implementation. It selects appropriate tools — pandas for small data, PySpark for large, dbt for warehouse transforms — and includes proper error handling, retry logic, and checkpoint patterns so the pipeline can be safely re-run without double-processing.

For Airflow pipelines, it generates a complete DAG with task dependencies correctly wired, sensors for upstream availability, SLA settings, and alerting on failure. For dbt, it produces models with the correct materialization strategy (table vs incremental), tests for primary key uniqueness and referential integrity, and documentation YAML. Every pipeline includes observability hooks — logging at each stage boundary so failures are easy to trace.

How to install

npx skills add user/data-pipeline

How to use

Design a data pipeline: pull orders from MySQL daily, compute revenue by product, load to BigQuery

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

default_args = {'retries': 3, 'retry_delay': timedelta(minutes=5)}

with DAG('orders_to_bigquery', default_args=default_args,
        schedule_interval='@daily', start_date=datetime(2026, 1, 1)) as dag:
  extract = PythonOperator(task_id='extract_orders', ...)
  transform = PythonOperator(task_id='compute_revenue', ...)
  load = PythonOperator(task_id='load_to_bq', ...)
  extract >> transform >> load

Configuration

# Switch to dbt
Generate a dbt incremental model for this transform instead

# Add data quality checks
Add Great Expectations data quality checks at each stage

# Scale up
Rewrite this for 100M rows using PySpark on Databricks

Related skills

Pair Data Pipeline Architect with CSV Data Analyzer to understand your source data before designing the pipeline, and SQL Query Builder to optimise the transformation queries.