Implementation Guides

Data Quality Framework for AI Projects

A practical guide to establishing data quality standards for AI and ML projects. Covers data profiling, validation rules, monitoring, and governance best practices.

Gabriele Masetti · 2026-08-10

Why "data quality" needs a framework, not a checklist

Most AI teams discover data quality problems the expensive way: a model's accuracy drops in production, someone traces it back three weeks later to a schema change in an upstream source table, and by then bad predictions have already shipped. A data quality framework is simply the set of dimensions you check, the tools that check them, and the points in your pipeline where checks run automatically instead of relying on someone noticing.

For a beginner-friendly rollout, you don't need every tool in this guide on day one. You need to know the six dimensions that matter, pick one validation tool that fits your stack, and put checks at the two or three places in your pipeline where bad data actually enters — ingestion, transformation, and pre-training.

The six core dimensions

Every data quality framework, from enterprise data governance programs down to a single dbt project, is built on the same six dimensions. Define them concretely for your own data before picking tools.

• Completeness — are required fields populated? Example: what percentage of customer_id or email values are null in a table that should never have nulls there.
• Accuracy — does the data reflect reality? Example: a birth_date of 2087 or a price of -50 is complete but not accurate.
• Consistency — do the same facts agree across systems and over time? Example: a customer's country field says "US" in the CRM and "United States" in the warehouse.
• Timeliness — is the data fresh enough to be useful? Example: a feature store table that's supposed to update hourly but hasn't refreshed in two days.
• Validity — does data conform to defined formats, types, and ranges? Example: an email column with values that aren't valid email strings, or a categorical status column containing a value outside the accepted set.
• Uniqueness — are there duplicate records where there shouldn't be? Example: the same order_id appearing twice after a failed retry in an ETL job.

In practice, most teams start by instrumenting completeness, validity, and uniqueness — they're the cheapest to check and catch the majority of pipeline bugs — then add consistency and timeliness once they have cross-system or streaming data, and accuracy last, since it usually requires business-specific rules or reference data.

Where checks belong in the pipeline

A useful mental model: data quality is not one tool, it's a layer that shows up at three stages.

1. At ingestion — validate incoming data against a schema or contract before it lands anywhere. This is where you catch upstream schema changes and malformed records cheaply, before they propagate.
2. In the transformation/warehouse layer — test the tables your analytics and feature pipelines actually read from. This is where dbt tests and SQL-based checks live.
3. In Python-based ML pipelines — validate dataframes going into feature engineering or training, and validate labels before they're used for training. This is where Great Expectations, Pandera, and label-QA metrics live.

Each stage benefits from a different tool, and most mature teams run more than one.

Tool selection for beginners

Two of those six changed owner in 2026 without changing their APIs. Fivetran and dbt Labs completed their merger on 1 June 2026, and Fivetran announced on 13 May 2026 that it would become steward of the Great Expectations open-source community and the GX Core project. Both remain open source, and nothing in the code below was invalidated by either move.

You do not need all six. A common beginner path: dbt tests for warehouse tables (because you likely already have dbt), Pandera or Great Expectations for the Python pipeline that builds your training set, and Evidently once the model is in production and you need drift monitoring.

Getting started with dbt tests

If your data already flows through dbt, this is the lowest-effort starting point. dbt tests are SQL queries that return failing rows — zero rows means the test passes. dbt Core ships four generic tests out of the box: not_null, unique, accepted_values, and relationships.

Those four survived the biggest change the project has had. On 1 June 2026, the day the Fivetran merger closed, dbt Labs released dbt Core v2.0: the Rust-based Fusion engine, published to the dbt-core repository under an Apache 2.0 licence, now serves as the shared runtime for both the open-source and the commercial distributions. The engine underneath changed; the schema.yml test syntax below did not.

# models/schema.yml
models:
  - name: customers
    columns:
      - name: customer_id
        tests:
          - unique
          - not_null
      - name: signup_country
        tests:
          - accepted_values:
              values: ['US', 'CA', 'GB', 'DE', 'FR']
      - name: plan_id
        tests:
          - relationships:
              to: ref('plans')
              field: plan_id

Run dbt test in CI on every pull request and on a schedule against production tables. When you outgrow the four built-in tests, add the dbt-expectations package for freshness checks, distribution checks, and multi-column logic.

Validating dataframes with Pandera

Pandera lets you define a schema once, then validate pandas, polars, or PySpark dataframes against it wherever they appear in your code. It's a natural fit right before a feature-engineering or training step.

Import from the backend-specific module, not the top level. Since v0.24, defining a pandas schema through import pandera as pa raises a FutureWarning and points you at import pandera.pandas as pa; the equivalent modules are pandera.polars and pandera.pyspark. Old tutorials still show the top-level import, and copying one is the fastest way to fill your logs with deprecation warnings.

import pandera.pandas as pa
from pandera.pandas import Column, Check, DataFrameSchema

training_data_schema = DataFrameSchema({
    "user_id": Column(int, Check.greater_than(0), nullable=False),
    "age": Column(int, Check.in_range(0, 120)),
    "signup_date": Column(pa.DateTime),
    "churned": Column(int, Check.isin([0, 1])),
    "email": Column(str, Check.str_matches(r"^[^@]+@[^@]+\.[^@]+$")),
}, strict=True)

# Raises pandera.errors.SchemaError with a readable diff if validation fails
validated_df = training_data_schema.validate(df, lazy=True)

Setting lazy=True collects every violation into a single error report instead of stopping at the first one — useful when you want to see the full scope of a data quality problem in one run rather than fixing issues one at a time.

Declarative validation with Great Expectations

Great Expectations (distributed as GX Core, still on the 1.x line) is a heavier-weight but more auditable option: you define "Expectations" (single assertions), group them into an Expectation Suite, tie that suite to a Batch of data via a Validation Definition, and run it through a Checkpoint that produces Data Docs — an auto-generated HTML report of what passed and failed. It works against pandas, Spark, SQLAlchemy-backed warehouses, and most common data sources.

import great_expectations as gx

context = gx.get_context()

data_source = context.data_sources.add_pandas("pandas_datasource")
data_asset = data_source.add_dataframe_asset(name="training_data")
batch_definition = data_asset.add_batch_definition_whole_dataframe("training_data_batch")

suite = context.suites.add(
    gx.core.expectation_suite.ExpectationSuite(name="training_data_suite")
)
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="user_id"))
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(column="age", min_value=0, max_value=120)
)
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="user_id"))

validation_definition = context.validation_definitions.add(
    gx.core.validation_definition.ValidationDefinition(
        name="training_data_validation", data=batch_definition, suite=suite
    )
)
checkpoint = context.checkpoints.add(
    gx.checkpoint.checkpoint.Checkpoint(
        name="training_data_checkpoint", validation_definitions=[validation_definition]
    )
)

result = checkpoint.run(batch_parameters={"dataframe": df})
if not result.success:
    raise ValueError("Data quality validation failed — see Data Docs for details")

The advantage over ad hoc assertions is the artifact: Data Docs give non-engineers (a data steward, a product manager) a report they can read without touching code, and the Suite becomes a versioned, reviewable definition of "what good data looks like" for that dataset.

Data contracts: catching problems before they're your problem

Everything above validates data after it arrives. A data contract shifts the check upstream: it's an explicit, versioned agreement between the team producing a dataset and the teams consuming it, specifying the schema, quality rules, and update cadence the producer commits to. The Open Data Contract Standard (ODCS), maintained by the Bitol project under the Linux Foundation, is a YAML-based specification for exactly this — schema, data quality rules, SLAs, and ownership in one document that both sides can diff and version in git. Pin the version you write against: v3.2.0, released in September 2026, is the current one.

For streaming data, the equivalent mechanism is a schema registry (Confluent Schema Registry is the common choice for Kafka), which enforces schema compatibility rules at write time and rejects producers that would break consumers. The principle in both cases: instead of discovering a breaking schema change when your validation suite fails downstream, the producer is contractually blocked from making it — or at least required to version it — before it ships.

Schema and data drift monitoring in production

Data contracts and validation suites catch problems at write time. Once a model is live, you also need to watch for gradual shifts that no single check will catch on its own:

• Schema drift — a column gets renamed, a type changes from integer to string, or a field silently starts arriving as null more often. This is best caught by re-running the same schema validation (Pandera, GX, or dbt tests) on every batch, not just once at build time.
• Data drift — the statistical distribution of incoming features shifts even though the schema is unchanged (e.g., customer age distribution shifts after a marketing campaign targets a new segment). Evidently, an open-source Python library, is built for this: it compares a reference dataset against current production data using statistical tests (Kolmogorov-Smirnov for numeric features, chi-squared for categorical ones on smaller datasets, and distance metrics like Wasserstein distance on larger ones) and can output HTML reports or feed metrics into Grafana or Airflow.

A practical setup: run your schema/validity checks (dbt tests, Pandera, or GX) on every pipeline run as a hard gate that can block a deploy, and run drift detection (Evidently) on a schedule as a softer signal that triggers an alert for a human to investigate, since drift doesn't always mean something broke — it can also mean the world changed.

Labeling quality: the dimension tabular tools miss

If your project involves human-labeled training data — annotated images, labeled text, human feedback — the six dimensions above still apply, but the primary quality signal is different: inter-annotator agreement (IAA). When multiple people label the same examples, agreement measures how consistently they interpret the labeling task, most commonly using Cohen's Kappa (two annotators) or Krippendorff's Alpha (more than two annotators, or missing labels).

Two things worth knowing as a beginner here:

• Low agreement usually means your labeling guidelines are ambiguous, not that your annotators are careless. Fix the guideline and re-measure before assuming you need better annotators.
• High agreement is necessary but not sufficient — if all annotators consistently apply the same wrong rule, agreement will look great while the labels are still bad. Combine IAA with periodic expert review of a random sample, not IAA alone.

Practically, run IAA calculation on a 5-10% overlap sample of every labeling batch, and treat a sustained drop in agreement the same way you'd treat a failed schema test: block the batch from entering training data until it's investigated.

A beginner's rollout plan

If you're introducing this at an organization with no existing data quality tooling, a realistic sequence looks like:

1. Pick 3-5 tables or datasets that feed your highest-stakes model and write not_null, unique, and accepted_values tests for their critical columns (dbt tests if you use dbt; Pandera or GX otherwise).
2. Wire those tests into CI so a pull request that would break them fails before merge, and into your scheduled pipeline runs so production data is checked continuously, not just at build time.
3. Add a data contract (even a lightweight, hand-written YAML file following the ODCS structure) for your most fragile upstream dependency — the one that has broken your pipeline before.
4. Once the model is in production, add drift monitoring with Evidently on the features that matter most, and set an alert threshold rather than trying to review every report manually.
5. If labeling is part of your pipeline, start measuring IAA on new labeling batches before you use them for training, not after a model trained on them underperforms.

None of this needs to be built at once. The teams that succeed with data quality frameworks are the ones that get one or two dimensions checked automatically and expand from there — not the ones that try to instrument all six dimensions across every table before shipping anything.