Data Engineering project ideas

100 projects, from a first build to a final-year capstone. Each lists the skills it teaches and the tools it needs; times and costs are estimates.

Beginner projects

• CSV Quality Profiler — Profile a CSV for types, nulls, duplicates and outliers and produce a quality report. Skills: data profiling, quality checks. Tools: Python, pandas. Data lab, Up to 15 h, Software only (free tools).
• Spreadsheet to Database Loader — Load spreadsheet data into a relational database with typed columns and keys. Skills: data loading, schema design. Tools: Python, SQLite. Workflow, Up to 15 h, Software only (free tools).
• SQL Query Practice Dataset — Write SQL queries over a sample sales database and check results. Skills: SQL, joins, aggregation. Tools: SQLite. Data lab, Up to 15 h, Software only (free tools).
• JSON to Table Flattener — Flatten nested JSON records into tables with clear column names. Skills: data transformation, JSON. Tools: Python. Workflow, Up to 15 h, Software only (free tools).
• Weather API Daily Collector — Fetch daily weather data on a schedule and store it with timestamps. Skills: API ingestion, scheduling. Tools: Python, cron. Workflow, Up to 15 h, Software only (free tools).
• Data Dictionary Builder — Document tables, columns, types and meanings for a dataset. Skills: metadata, documentation. Tools: spreadsheet. Workflow, Up to 15 h, Software only (free tools).
• Entity Relationship Diagram Designer — Design an ER diagram for a small business and generate tables. Skills: data modelling, ER diagrams. Tools: diagram tool. CAD / design, Up to 15 h, Software only (free tools).
• File Format Size Comparison — Compare CSV, JSON and Parquet sizes and read speeds. Skills: file formats, benchmarking. Tools: Python. Data lab, Up to 15 h, Software only (free tools).
• Duplicate Record Finder — Find exact and fuzzy duplicate customer records. Skills: deduplication, fuzzy matching. Tools: Python. Data lab, Up to 15 h, Software only (free tools).
• Data Pipeline Diagram Explainer — Explain extract, transform and load stages with an interactive diagram. Skills: ETL concepts, visual explanation. Tools: web app. Visualizer, Up to 15 h, Software only (free tools).
• Log File Parser — Parse web server logs into structured fields and count status codes. Skills: log parsing, regex. Tools: Python. Data lab, Up to 15 h, Software only (free tools).
• Date and Time Zone Normaliser — Normalise mixed date formats and time zones to UTC. Skills: data cleaning, time zones. Tools: Python. Workflow, Up to 15 h, Software only (free tools).
• Simple Dashboard From SQL — Build a dashboard from SQL aggregates of sample data. Skills: SQL, visualisation. Tools: Metabase/Python. Dashboard, Up to 15 h, Software only (free tools).
• Database Indexing Experiment — Time queries before and after adding indexes. Skills: indexing, query performance. Tools: PostgreSQL. Data lab, Up to 15 h, Software only (free tools).
• Data Validation Rules Checklist — Write validation rules for a dataset and test them. Skills: validation, constraints. Tools: Python. Workflow, Up to 15 h, Software only (free tools).
• Open Data Portal Explorer — Find and download open datasets and record their licences. Skills: data sourcing, licensing. Tools: web. Workflow, Up to 15 h, Software only (free tools).
• Normalisation Exercise Kit — Normalise a flat table to third normal form with checks. Skills: normalisation, relational design. Tools: SQL. Data lab, Up to 15 h, Software only (free tools).
• Web Table Scraper With Etiquette — Extract tables from permitted pages respecting robots rules and rate limits. Skills: web extraction, ethics. Tools: Python. Workflow, Up to 15 h, Software only (free tools).
• Backup and Restore Drill — Back up a database, restore it and verify row counts. Skills: backups, recovery. Tools: PostgreSQL. Workflow, Up to 15 h, Software only (free tools).
• Spreadsheet Formula to SQL Translator — Rewrite spreadsheet calculations as SQL views. Skills: SQL views, transformation. Tools: SQL. Data lab, Up to 15 h, Software only (free tools).
• Sample Data Generator — Generate realistic fake data for testing pipelines. Skills: test data generation, data analysis. Tools: Python, Faker. Data lab, Up to 15 h, Software only (free tools).
• Data Types and Encoding Explainer — Show how numbers, text and encodings are stored. Skills: data types, encodings. Tools: web app. Visualizer, Up to 15 h, Software only (free tools).
• Column Statistics Visualizer — Plot histograms and summary stats for each column. Skills: exploratory analysis, visual explanation. Tools: Python. Visualizer, Up to 15 h, Software only (free tools).
• Batch Job Scheduler With Cron — Schedule scripts and log successes and failures. Skills: scheduling, logging. Tools: cron. Workflow, Up to 15 h, Software only (free tools).
• Data Privacy Basics Checklist — Identify personal data fields and decide how to protect them. Skills: privacy awareness, process documentation. Tools: template. Workflow, Up to 15 h, Software only (free tools).

Intermediate projects

• Incremental ETL Pipeline — Load only new and changed records using watermarks and idempotent upserts. Skills: incremental loading, idempotency. Tools: Python, PostgreSQL. Workflow, 15–45 h, Software only (free tools).
• Data Contract Validator — Validate producer data against schema and quality contracts before loading. Skills: data contracts, validation. Tools: Python, JSON Schema. Workflow, 15–45 h, Software only (free tools).
• CDC Replication Demo — Capture database changes and replicate them to another store. Skills: change data capture, numerical modelling. Tools: Debezium/Postgres. Simulator, 15–45 h, Software only (free tools).
• Streaming Event Processor — Process events from a stream with windows and aggregations. Skills: stream processing, windows. Tools: Kafka, Python. Simulator, 15–45 h, Software only (free tools).
• Schema Evolution Simulator — Apply schema changes and test reader and writer compatibility. Skills: schema evolution, Avro. Tools: Python. Simulator, 15–45 h, Software only (free tools).
• Data Lineage Graph Explorer — Build a lineage graph from SQL and show upstream and downstream tables. Skills: lineage, SQL parsing. Tools: Python. Visualizer, 15–45 h, Software only (free tools).
• Warehouse Cost Monitor — Track query costs by user and table and flag expensive patterns. Skills: cost monitoring, FinOps. Tools: dashboard. Dashboard, 15–45 h, $30–150.
• PII Detection and Masking Pipeline — Detect personal data in columns and mask or tokenise it. Skills: PII detection, masking. Tools: Python. Workflow, 15–45 h, Software only (free tools).
• Slowly Changing Dimension Lab — Implement SCD types 1, 2 and 3 and query history. Skills: dimensional modelling, SCD. Tools: SQL. Data lab, 15–45 h, Software only (free tools).
• Batch Backfill Orchestrator — Backfill partitions safely with retries and progress tracking. Skills: orchestration, backfills. Tools: Airflow/Dagster. Workflow, 15–45 h, Software only (free tools).
• Star Schema Warehouse Design — Design fact and dimension tables for a retail warehouse. Skills: star schemas, modelling. Tools: SQL. CAD / design, 15–45 h, Software only (free tools).
• dbt Transformation Project — Build tested SQL models with dbt and generate docs. Skills: dbt, analytics engineering. Tools: dbt. Workflow, 15–45 h, Software only (free tools).
• Airflow DAG Pipeline — Orchestrate extract, transform and load tasks with dependencies. Skills: orchestration, DAGs. Tools: Airflow. Workflow, 15–45 h, Software only (free tools).
• Partitioning Strategy Study — Compare partitioning schemes on query speed and cost. Skills: partitioning, performance. Tools: Spark/DuckDB. Data lab, 15–45 h, Software only (free tools).
• Data Quality Test Suite — Write expectation tests and run them in CI. Skills: data testing, process documentation. Tools: Great Expectations. Workflow, 15–45 h, Software only (free tools).
• Object Storage Data Lake Setup — Organise raw, clean and curated zones in object storage. Skills: data lakes, layout. Tools: MinIO/S3. Workflow, 15–45 h, Software only (free tools).
• API Pagination Ingestion Service — Ingest paginated APIs with rate limits and retries. Skills: API ingestion, resilience. Tools: Python. Workflow, 15–45 h, Software only (free tools).
• Columnar Query Engine Experiment — Query Parquet files with DuckDB and compare engines. Skills: columnar analytics, data analysis. Tools: DuckDB. Data lab, 15–45 h, Software only (free tools).
• Data Catalog Search Prototype — Index datasets with tags and descriptions for search. Skills: data catalogs, search. Tools: Python. Workflow, 15–45 h, Software only (free tools).
• IoT Sensor Ingestion Pipeline — Ingest MQTT sensor messages into a time-series store. Skills: IoT ingestion, time series. Tools: MQTT, InfluxDB. Dashboard, 15–45 h, Software only (free tools).
• SQL Query Plan Explainer — Read and visualise query plans to find bottlenecks. Skills: query plans, optimisation. Tools: PostgreSQL. Visualizer, 15–45 h, Software only (free tools).
• Event Schema Registry — Register event schemas and reject incompatible versions. Skills: schema registries, process documentation. Tools: Python. Workflow, 15–45 h, Software only (free tools).
• Data Freshness Alerting — Alert when tables miss expected update times. Skills: freshness SLAs, alerting. Tools: Python. Dashboard, 15–45 h, Software only (free tools).
• ELT With Cloud Warehouse Emulator — Load raw data and transform inside the warehouse. Skills: ELT patterns, process documentation. Tools: DuckDB. Workflow, 15–45 h, Software only (free tools).
• Data Pipeline Unit Testing — Unit-test transformation functions with fixtures. Skills: testing, pipelines. Tools: pytest. Workflow, 15–45 h, Software only (free tools).

Advanced projects

• Lakehouse Medallion Pipeline — Build bronze, silver and gold layers with table formats and quality gates. Skills: lakehouse, table formats. Tools: Spark, Delta/Iceberg. Workflow, 45–100 h, Software only (free tools).
• Data Observability Scorecard — Score tables on freshness, volume, schema and distribution health. Skills: data observability, KPI design. Tools: Python, dashboard. Dashboard, 45–100 h, $30–150.
• Kafka Dead-Letter Queue Monitor — Route failed messages to a DLQ and track reprocessing. Skills: DLQ patterns, Kafka. Tools: Kafka. Dashboard, 45–100 h, Software only (free tools).
• Geospatial Data Pipeline — Ingest, clean and index geospatial data for map queries. Skills: geospatial processing, process documentation. Tools: PostGIS, Python. Workflow, 45–100 h, Software only (free tools).
• Feature Store Prototype — Serve consistent features for training and online inference. Skills: feature stores, consistency. Tools: Python, Redis. Workflow, 45–100 h, Software only (free tools).
• Data Retention Policy Engine — Apply retention and deletion rules with audit logs. Skills: retention, governance. Tools: Python. Workflow, 45–100 h, Software only (free tools).
• Distributed Deduplication Service — Deduplicate high-volume records with keys and bloom filters. Skills: distributed dedup, bloom filters. Tools: Spark. Simulator, 45–100 h, Software only (free tools).
• Exactly-Once Stream Processing Lab — Compare at-least-once and exactly-once semantics with failures. Skills: delivery semantics, numerical modelling. Tools: Kafka, Flink. Simulator, 45–100 h, Software only (free tools).
• Spark Job Performance Tuning — Tune shuffles, partitions and joins in Spark jobs. Skills: Spark tuning, data analysis. Tools: Spark. Data lab, 45–100 h, Software only (free tools).
• Data Vault Modelling Project — Model hubs, links and satellites for an integration layer. Skills: data vault modelling, technical drawing. Tools: SQL. CAD / design, 45–100 h, Software only (free tools).
• Reverse ETL Sync Service — Sync warehouse data back to operational tools safely. Skills: reverse ETL, process documentation. Tools: Python. Workflow, 45–100 h, Software only (free tools).
• Time-Travel Table Audit — Query historical table versions for audits and rollback. Skills: table versioning, data analysis. Tools: Delta/Iceberg. Data lab, 45–100 h, Software only (free tools).
• Data Access Control Matrix — Implement row and column security by role. Skills: access control, governance. Tools: PostgreSQL. Workflow, 45–100 h, Software only (free tools).
• Lambda vs Kappa Architecture Study — Compare batch-plus-stream and stream-only designs. Skills: architecture trade-offs, numerical modelling. Tools: diagrams, prototypes. Simulator, 45–100 h, Software only (free tools).
• Data Pipeline Cost Optimiser — Reduce compute and storage costs with scheduling and formats. Skills: cost optimisation, KPI design. Tools: Python. Dashboard, 45–100 h, Software only (free tools).
• Entity Resolution Pipeline — Match records across sources with blocking and scoring. Skills: entity resolution, data analysis. Tools: Python. Data lab, 45–100 h, Software only (free tools).
• Real-Time Analytics Dashboard — Serve sub-second metrics from streaming aggregates. Skills: real-time OLAP, KPI design. Tools: ClickHouse. Dashboard, 45–100 h, Software only (free tools).
• Metadata-Driven Ingestion Framework — Configure new sources by metadata instead of code. Skills: framework design, process documentation. Tools: Python. Workflow, 45–100 h, Software only (free tools).
• Data SLA Incident Runbook — Write runbooks and post-mortems for data incidents. Skills: incident management, process documentation. Tools: template. Workflow, 45–100 h, Software only (free tools).
• Graph Database Modelling Study — Model relationships in a graph database and compare queries. Skills: graph modelling, data analysis. Tools: Neo4j. Data lab, 45–100 h, Software only (free tools).
• Search Index Pipeline — Index documents into a search engine with updates. Skills: search indexing, process documentation. Tools: OpenSearch. Workflow, 45–100 h, Software only (free tools).
• Data Anonymisation Utility Study — Measure utility loss after k-anonymity transformations. Skills: anonymisation, utility. Tools: Python. Data lab, 45–100 h, Software only (free tools).
• Container-Based Pipeline Deployment — Package pipelines in containers and deploy with configs. Skills: containers, deployment. Tools: Docker. Workflow, 45–100 h, Software only (free tools).
• Orchestrator Comparison Study — Compare Airflow, Dagster and Prefect on one pipeline. Skills: orchestration comparison, data analysis. Tools: three orchestrators. Data lab, 45–100 h, Software only (free tools).
• Semi-Structured Data Modelling — Model JSON events for analytics with flattening strategies. Skills: semi-structured modelling, data analysis. Tools: DuckDB. Data lab, 45–100 h, Software only (free tools).

Expert projects

• Real-Time Fraud Feature Pipeline — Compute streaming features for fraud scoring with low latency. Skills: streaming features, latency. Tools: Kafka, Flink. Simulator, 100 h+, Software only (free tools).
• Privacy-Aware Data Sharing Layer — Share data with partners using policies, masking and audit. Skills: data sharing, privacy. Tools: Python. Workflow, 100 h+, Software only (free tools).
• Query Workload Optimizer — Analyse query logs and recommend indexes and materialised views. Skills: workload analysis, optimisation. Tools: Python, SQL. Data lab, 100 h+, Software only (free tools).
• Multi-Cloud Ingestion Framework — Ingest from multiple clouds with portable connectors. Skills: multi-cloud, connectors. Tools: Python. Workflow, 100 h+, Software only (free tools).
• Data Mesh Domain Catalog — Publish domain data products with owners, contracts and SLAs. Skills: data mesh, data products. Tools: database app. Workflow, 100 h+, Software only (free tools).
• Reproducible Analytics Pipeline — Pin data, code and environments so results can be reproduced. Skills: reproducibility, process documentation. Tools: DVC, containers. Workflow, 100 h+, Software only (free tools).
• Self-Healing Data Platform — Detect failures and auto-retry, quarantine or roll back. Skills: resilience, automation. Tools: Python. Simulator, 100 h+, Software only (free tools).
• Distributed Query Engine From Scratch — Build a tiny engine that scans, filters and aggregates across workers. Skills: distributed query processing, numerical modelling. Tools: Python. Simulator, 100 h+, Software only (free tools).
• Streaming Join Strategies Study — Compare stream-stream and stream-table joins under lateness. Skills: stream joins, watermarks. Tools: Flink. Simulator, 100 h+, Software only (free tools).
• Data Platform Digital Twin — Simulate pipeline loads and failures to plan capacity. Skills: capacity planning, simulation. Tools: Python. Digital twin, 100 h+, Software only (free tools).
• Vector Database Ingestion Pipeline — Embed and index documents with update and delete handling. Skills: vector databases, process documentation. Tools: Python. Workflow, 100 h+, Software only (free tools).
• Open Table Format Compaction Service — Compact small files and measure query gains. Skills: table maintenance, data analysis. Tools: Spark. Data lab, 100 h+, Software only (free tools).
• Column-Level Lineage Extraction — Parse SQL to trace lineage at column level. Skills: SQL parsing, lineage. Tools: Python. Visualizer, 100 h+, Software only (free tools).
• Data Quality Anomaly Detection With ML — Detect unusual metrics in table statistics automatically. Skills: ML monitoring, data analysis. Tools: Python. Data lab, 100 h+, Software only (free tools).
• Federated Query Over Sources — Query across databases and files without moving data. Skills: federated queries, data analysis. Tools: Trino. Data lab, 100 h+, Software only (free tools).
• Data Governance Policy as Code — Express governance rules as code and test them. Skills: policy as code, process documentation. Tools: Python. Workflow, 100 h+, Software only (free tools).
• Event Sourcing Analytics Store — Rebuild state from events and support time-travel queries. Skills: event sourcing, numerical modelling. Tools: Python. Simulator, 100 h+, Software only (free tools).
• High-Throughput Ingestion Benchmark — Benchmark ingestion throughput and tune batching. Skills: benchmarking, throughput. Tools: Kafka. Data lab, 100 h+, Software only (free tools).
• Synthetic Data Platform for Testing — Generate privacy-safe test data matching production schemas. Skills: synthetic data, testing. Tools: Python. Workflow, 100 h+, Software only (free tools).
• Data Product Usage Analytics — Measure who uses which data products and how. Skills: usage analytics, KPI design. Tools: dashboard. Dashboard, 100 h+, $150+.
• Carbon-Aware Pipeline Scheduler — Shift batch jobs to low-carbon grid hours. Skills: green computing, scheduling. Tools: Python. Simulator, 100 h+, Software only (free tools).
• Cross-Region Replication Study — Measure replication lag and consistency across regions. Skills: replication, consistency. Tools: database. Data lab, 100 h+, Software only (free tools).
• LLM-Assisted SQL Generation Validator — Validate generated SQL against tests and permissions. Skills: SQL validation, LLMs. Tools: Python. Workflow, 100 h+, Software only (free tools).
• Unified Batch and Stream Pipeline — Run one codebase in batch and streaming modes. Skills: unified processing, process documentation. Tools: Apache Beam. Workflow, 100 h+, Software only (free tools).
• Data Platform Security Hardening — Apply encryption, least privilege and audit logging defensively. Skills: platform security, defence. Tools: checklist, configs. Workflow, 100 h+, Software only (free tools).

Project ideas for every branch · Data Engineering roadmap