Storage & Distributed Compute

Warehouses, lakes and Spark. Columnar formats, partitioning, lakehouse table formats, and distributed processing with MapReduce and Spark.

• Storage: Warehouses, Lakes, Lakehouses & File Formats — Where and how data is stored for analytics. Learn row vs columnar storage, file formats (CSV, JSON, Avro, Parquet, ORC), compression and encoding (dictionary, run-length, delta), partitioning, clustering and pruning, the small-files problem, data warehouses (BigQuery, Snowflake, Redshift), data lakes on object storage, open table formats (Iceberg, Delta Lake, Hudi) with ACID transactions and time travel, lakehouse architecture, caching and materialised views, and storage cost optimisation — with Python comparing row and column scans, run-length encoding and partition pruning.
• Distributed Processing: MapReduce, Spark & Big Data — Processing data too big for one machine. Learn why we distribute (volume, parallelism), horizontal scaling and Amdahl's law, MapReduce (map, shuffle, reduce), Apache Spark (driver/executors, RDDs and DataFrames, lazy evaluation, DAG and stages, narrow vs wide transformations, the shuffle), partitioning and data skew, broadcast joins, caching, fault tolerance via lineage, cluster sizing, and alternatives (Dask, Ray, Flink, BigQuery, DuckDB) — with Python simulating MapReduce word count with hash partitioning and skew measurement.