Last year around April, there was a production issue in our data engineering system and we had to roll back a recently merged pull request. This pull request had minor code changes and was reviewed and approved by me and one more person on my team. The change was made in the code of one of the upstream tables of a pipeline, which resulted in a schema violation in a downstream table. The downstream table was part of a separate pipeline, hence it was missed by the data engineer who raised the pull request.
The change was quite small - and therefore tempting to approve quickly and be done with. Ideally, I should have looked at the pipeline runs in the development environment and verified all the downstream tables’ schema contracts and data. But practically speaking, this is a cumbersome and time-consuming task. Therefore, on busy days this usually gets skipped.
Post that incident, whenever I reviewed PRs, I tried to look at the changes at the data level as well - at least whenever time permitted. I took more time to review (which was of course a bit frustrating both for the PR author and the reviewer). This habit turned into a project over time. I have recently open-sourced it to help others in the data engineering community. This article covers that project.
Software teams have a safety net. Data teams mostly don’t.
In software engineering, a pull request triggers a build that runs a full suite of tests to ensure every component is working correctly and nothing is breaking due to the current set of changes. The build and the test results give an objective signal to the reviewer about the changes. It also helps newcomers to the team, who may not yet know all the components fully, to author their changes or review others' changes properly.
On the other hand, PRs raised in data engineering teams often have gaps in this process. A data engineer makes the code changes, creates/updates/deletes tables in the dev environment, tests the changes by running the pipelines and running other data quality tests. The first gap is here itself - governance. Unlike software engineering PRs, where there are coded tests for almost every related component that are run during the build, there are hardly any such tests set up to run automatically before merging any changes to production. There are usually no written schema contracts that the tests are run against. It depends on the practices followed in the team and the goodwill of the developer as to how thoroughly these tests are done.
The second major gap is lack of automation. The pull request itself does not trigger any build as such - one that tests all the code, pipeline runs, impact on downstream tables, etc. As I mentioned at the beginning of the article, the reviewer has to check the runs manually in the dev branch and, at times, run the tests to ensure nothing is breaking downstream.
The third gap is lack of visibility for new team members who do not have sufficient experience or knowledge about the team's data estate. They may miss out on testing some of the tables while making the changes or reviewing others' changes.
Most data teams have data quality checks in place to identify any issues in production and minimise the risk of corrupting data in any of the downstream tables. These are fine, but they catch the issues in production, which is the fourth and final gap - lack of proactiveness. Although we catch the issues in production through these checks, in most cases the damage is partly already done - be it delayed runtimes, failure to meet SLAs or simply wasted compute in rerunning some of the pipelines.
DEBIT - Data Engineering Build Impact Testing
In order to close the above gaps, I built DEBIT. On every pull request raised in our team, this framework runs the tests needed automatically at both the code level and the data level. It identifies the tables impacted, checks their contracts mentioned in YAML files, figures out what the downstream and upstream tables for these impacted tables are, and then runs the tests on the actual data in dev environments. Finally, it stores those results and displays them in the PR build log for the reviewer to inspect.
It targets Microsoft Fabric and lakehouse workflows, but the core is plain Python with no cloud dependency. The repo includes templates for GitHub Actions and Azure DevOps.
DEBIT is open source under the MIT license: github.com/pratap-is-here/debit
An Important Note
The framework does not replace any testing that the PR author must do. It supports it by bringing governance to the whole process of moving any changes from dev to prod, by ensuring increased thoroughness and efficiency as well as reducing the review workload on the reviewers. So the PR author is still strongly recommended to do the testing on appropriate data and, if needed, copy some of the data from prod to dev before testing.
DEBIT assumes the developer has already tested in dev against the latest prod data. If they haven’t, the validations fail, and it becomes visible in the PR.
That's deliberate. DEBIT checks that testing happened. It doesn't replace it.
2 Phases
Every PR runs through 2 phases.
Phase 1 is an offline hard gate. It validates the table declarations in the PR template, reads the git diff to find the affected tables, checks the registry for completeness, cross-checks the lineage for consistency, expands the scope along the lineage graph, and verifies that every table in scope has a complete contract. It needs no cloud access.
Phase 2 is optional and live. It builds a contract payload from the table registry, triggers a Fabric notebook through the Jobs API, and runs row count, freshness, schema and custom checks. It then reads the results back from OneLake and prints a structured pass/fail report. By default it warns instead of blocking, and you can switch it to a hard gate.
I made Phase 1 the hard gate because it’s deterministic and runs offline. The live checks depend on a Fabric run, so I wanted them to start as warnings until a team trusts them.
Lineage decides the Scope
Testing everything on every PR is slow, and testing only the changed table misses the damage downstream. So the PR author chooses how far validation reaches.
Each table lists its direct downstream consumers in a YAML file, and upstream is worked out from that at runtime. The author picks one of four modes in the PR description.
In the diagram, Silver F is the table the PR touched. Affected only validates just F. Upstream adds everything F reads from, six tables in all. Downstream adds what reads from F, three tables. Full lineage covers both directions, eight tables. Silver G and Gold J sit beside F in the graph, not above or below it, so they stay out of scope.
Metadata over Code
All the rules live in YAML, not in hardcoded logic.\
Five core files are part of the metadata:
- subset_scope.yml: This file tells which tables, pipelines or notebooks are governed.
- table_registry.yml: This file covers the per table contracts.
- table_lineage.yml: This file mentions the downstream lineage.
- notebook_outputs.yml and pipeline_outputs.yml: These files tell which tables each notebook and pipeline produces.
Two more handle execution freshness and custom check notebooks.
Here’s what a contract looks like, trimmed from the synthetic example in the repo:
silver.fact_orders:
min_rows: 1000
freshness_hours: 24
freshness_column: order_date
schema:
required_columns:
- order_id
- customer_id
- order_amount
- order_date
not_null_columns:
- order_id
- customer_id
checks:
- name: order_amount_non_negative
sql: "SELECT COUNT(1) FROM {table} WHERE order_amount IS NOT NULL AND order_amount < 0"
expect_eq: 0
There's no Python to write. Changing a threshold is a visible edit in a PR, with a name and a reason attached. A newer engineer can read the file to see what healthy means for a table.
Rolling it out
You don’t have to cover everything at once.
You can start with a set of pilot tables and adjust the checks and framework based on your team’s requirements. Once you are confident with the updates, you can expand to more tables by simply updating the metadata files. Assets that are outside the scope will remain untouched.
Did it help?
On our team, it cut PR review time by roughly 40% and production incidents by roughly 25 to 30%.
Treat those as our experience, not a benchmark. Your tables, your review habits and your starting point will differ.
What it doesn’t do
- It assumes developers test in dev first. If the dev run is sloppy, DEBIT can only tell you that, not fix it.
- A contract is only as good as what you write into it. Too tight and the checks raise false alarms, too loose and they miss real problems, so the numbers need upkeep.
- The live checks only support Fabric today.
- YAML can’t express everything, so there’s a custom SQL check as an escape hatch. I’d rather keep those few, or the files turn into code.
- I built it for my own team, so expect rough edges.
Try it
You don't need cloud access to try it. The bundled example and the whole test suite run offline.
git clone https://github.com/pratap-is-here/debit
cd debit
pip install -r requirements.txt
python examples/run_validation_example.py
python -m pytest -q
The example walks one changed table through all four scope modes and prints a pass for each. The tests should report 114 passed.
DEBIT is open source under the MIT license: github.com/pratap-is-here/debit
If you review data PRs, I’d like to hear what’s missing. And if a field in the metadata files is awkward for your tables, open an issue with a small example of what you’d rather write. A real table that doesn’t fit is more useful to me than a star.
Views are my own.
Microsoft Employee