Data Quality, Observability & Governance

Making data trustworthy. Learn data quality dimensions (completeness, uniqueness, validity, consistency, accuracy, timeliness), writing checks and tests (dbt tests, Great Expectations, Soda), anomaly detection on metrics, data observability (freshness, volume, schema, distribution, lineage), SLAs/SLOs for data, incident management, data catalogs and lineage, metadata, access control and security, privacy (PII, masking, DPDP/GDPR), data governance roles (owners, stewards), and data contracts — with Python running completeness, uniqueness, validity and freshness checks.

Bad data is expensive: wrong dashboards drive wrong decisions, and ML models silently learn from errors. Most data incidents come from upstream changes nobody told the data team about.

Data quality engineering adds automated checks at every stage; observability detects anomalies; governance defines ownership, access and compliance so data can be trusted and used safely.

An analogy

Data quality checks are like food-safety inspections in a supply chain: you test ingredients when they arrive, monitor temperatures during processing, check the finished product, and trace any bad batch back to its source (lineage).

Quality dimensions

What 'good data' means:

Testing and observability

Tests encode known expectations (not null, unique, referential integrity, accepted values, row count within range). Observability detects unknown problems by monitoring freshness, volume, schema changes and value distributions over time with anomaly detection, then uses lineage to find root causes and impacted assets. Define data SLAs/SLOs (e.g. 'sales mart ready by 07:00 on 99% of days') and run incident processes like software teams do.

Metadata, catalogs and lineage

Catalogs (DataHub, OpenMetadata, Unity Catalog, Collibra) make data discoverable with descriptions, owners, tags, quality status and usage. Column-level lineage traces how fields flow from sources through transformations to dashboards and models — essential for impact analysis and audits. Data contracts formalise schemas and semantics between producers and consumers.

Governance, security and privacy

Governance defines data owners and stewards, policies, definitions (business glossary) and decision rights. Security: least-privilege access (RBAC/ABAC), row- and column-level security, encryption at rest and in transit, audit logs. Privacy: classify PII, minimise collection, mask/tokenise, pseudonymise for analytics, retention and deletion workflows to honour DPDP Act/GDPR rights, consent tracking. Good governance enables safe self-service rather than blocking it.

Formulas

Worked example: Completeness

Given: 1 000 000 rows, 12 000 null phone numbers

Find: Completeness.

Answer: 98.8%

Worked example: Volume anomaly

Given: Mean 1.0 M rows/day, std 50 k, today 700 k

Find: z-score.

Answer: −6 → alert (likely missing data)

Worked example: SLO

Given: Target 99% on-time; 3 late days in 90

Find: Met?

Answer: No — error budget exhausted

Automated data quality checks

from datetime import datetime
# Data quality checks: completeness, uniqueness, validity, freshness
rows = [
    {"id": 1, "email": "a@x.com", "age": 34, "updated": "2026-09-25T08:00"},
    {"id": 2, "email": None,      "age": 29, "updated": "2026-09-25T08:05"},
    {"id": 3, "email": "c@x.com", "age": -4, "updated": "2026-09-25T08:10"},
    {"id": 3, "email": "d@x.com", "age": 51, "updated": "2026-09-25T08:15"},
]
null_rate = sum(r["email"] is None for r in rows) / len(rows)
dupes = len(rows) - len({r["id"] for r in rows})
invalid_age = sum(not (0 <= r["age"] <= 120) for r in rows)
latest = max(datetime.fromisoformat(r["updated"]) for r in rows)
lag_h = (datetime.fromisoformat("2026-09-26T09:00") - latest).total_seconds() / 3600
checks = {"email null rate <= 5%": null_rate <= 0.05, "id unique": dupes == 0,
          "age in [0, 120]": invalid_age == 0, "fresh within 24 h": lag_h <= 24}
for name, ok in checks.items():
    print(f"{'PASS' if ok else 'FAIL'}  {name}")
print(f"null rate {null_rate:.0%}, duplicate ids {dupes}, invalid ages {invalid_age}, data lag {lag_h:.1f} h")
# FAIL  email null rate <= 5%
# FAIL  id unique
# FAIL  age in [0, 120]
# FAIL  fresh within 24 h
# null rate 25%, duplicate ids 1, invalid ages 1, data lag 24.8 h

Four rows, four failures: a missing email, a duplicated ID, an impossible age and data that's stale by 48 minutes beyond its 24 h SLA. In production these checks run inside the pipeline (dbt tests, Great Expectations) and block publishing ('write–audit–publish') or alert the owning team, with lineage showing which dashboards are affected.

Common mistakes

Applications

Key takeaways

Going further