data-engineering-data-pipeline

rmyndharis/antigravity-skillsData PlatformsAudit passedSnapshot 54f4427a52a7

Installs: 0

Summary

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

SKILL.md

Data Pipeline Architecture

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

Use this skill when

Working on data pipeline architecture tasks or workflows
Needing guidance, best practices, or checklists for data pipeline architecture

Do not use this skill when

The task is unrelated to data pipeline architecture
You need a different domain or tool outside this scope

Requirements

$ARGUMENTS

Core Capabilities

Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures
Implement batch and streaming data ingestion
Build workflow orchestration with Airflow/Prefect
Transform data using dbt and Spark
Manage Delta Lake/Iceberg storage with ACID transactions
Implement data quality frameworks (Great Expectations, dbt tests)
Monitor pipelines with CloudWatch/Prometheus/Grafana
Optimize costs through partitioning, lifecycle policies, and compute optimization

Instructions

1. Architecture Design

Assess: sources, volume, latency requirements, targets
Select pattern: ETL (transform before load), ELT (load then transform), Lambda (batch + speed layers), Kappa (stream-only), Lakehouse (unified)
Design flow: sources → ingestion → processing → storage → serving
Add observability touchpoints

2. Ingestion Implementation

Batch

Incremental loading with watermark columns
Retry logic with exponential backoff
Schema validation and dead letter queue for invalid records
Metadata tracking (_extracted_at, _source)

Streaming

Kafka consumers with exactly-once semantics

# Batch ingestion with validation
from batch_ingestion import BatchDataIngester
from storage.delta_lake_manager import DeltaLakeManager
from data_quality.expectations_suite import DataQualityFramework

ingester = BatchDataIngester(config={})

# Extract with incremental loading
df = ingester.extract_from_database(
    connection_string='postgresql://host:5432/db',
    query='SELECT * FROM orders',
    watermark_column='updated_at',
    last_watermark=last_run_timestamp
)

# Validate
schema = {'required_fields': ['id', 'user_id'], 'dtypes': {'id': 'int64'}}
df = ingester.validate_and_clean(df, schema)

# Data quality checks
dq = DataQualityFramework()
result = dq.validate_dataframe(df, suite_name='orders_suite', data_asset_name='orders')

# Write to Delta Lake
delta_mgr = DeltaLakeManager(storage_path='s3://lake')
delta_mgr.create_or_update_table(
    df=df,
    table_name='orders',
    partition_columns=['order_date'],
    mode='append'
)

# Save failed records
ingester.save_dead_letter_queue('s3://lake/dlq/orders')

data-engineering-data-pipeline

Summary

SKILL.md

Data Pipeline Architecture

Use this skill when

Do not use this skill when

Requirements

Core Capabilities

Instructions

1. Architecture Design

2. Ingestion Implementation

3. Orchestration

4. Transformation with dbt

5. Data Quality Framework

6. Storage Strategy

7. Monitoring & Cost Optimization

Example: Minimal Batch Pipeline

Output Deliverables

1. Architecture Documentation

2. Implementation Code

3. Configuration Files

4. Monitoring & Observability

5. Operations Guide

Success Criteria

Related skills