First Real-World Data Engineering Project on Fabric — Best Practices & Next Steps

Hi robertozsr ,

Great foundation — you're further along than most first Fabric projects I've seen. Let me take your questions one by one.

1. Best practices & priority steps to production-ready

The biggest gap I typically see at this stage is observability. Your pipeline works, but can you explain *why* it failed at 3am on a Saturday? Here's what I'd prioritize in order:

- Pipeline audit/control table — Log every run with start time, end time, rows read, rows written, status, and error messages. This is cheap to build and invaluable when debugging or proving to stakeholders that data is fresh.
- Failure alerting — Set up alerts on every pipeline activity. Fabric supports Teams and email notifications natively. Don't wait until something breaks silently for a week before you notice.
- Schema drift handling — Make your bronze ingestion defensive against source changes. A renamed or added column in Postgres shouldn't silently drop data — it should either be absorbed gracefully or fail loudly with a clear error.
- Idempotent pipelines — Make sure re-running a failed pipeline for the same day doesn't create duplicates. This matters more than people realize when you're firefighting at 7am.
- Gold layer incremental load — Your instinct to keep full overwrite for now is correct. When you're ready, Delta Lake's MERGE INTO handles the multi-source join complexity well. But don't touch this until your numbers are validated and the pipeline is stable.

2. What "going to production" means in Fabric

You're right that Fabric is workspace-centric, and yes — Deployment Pipelines (Dev → Test → Prod) is the standard approach for environment separation. Each stage maps to its own workspace with its own lakehouse, pipelines, notebooks, and semantic models.

A few things that trip people up with this feature:

- Deployment pipelines promote artifact *definitions* (pipeline structure, notebook code, semantic model schema), not data. Each workspace has its own lakehouse with its own data, populated by its own pipeline runs.
- You need to parameterize your connections — especially your Postgres source. Use deployment rules to swap connection strings per environment so Dev points at a staging database and Prod points at the real one.
- Lakehouse bindings also need to be configured per stage, so pipelines in Prod reference the Prod lakehouse, not the Dev one.
- Set this up early, even if your Test workspace is empty for now. It forces good habits around what's committed to Git versus what's ad-hoc, and avoids the painful "we've been building everything in Prod" conversation later.

In short: "going to production" in Fabric means having a dedicated Prod workspace that receives promoted, tested artifacts through the deployment pipeline, with its own connections and data — not just the workspace you've been developing in.

3. Automated layer-by-layer validation

There's no single native data quality feature in Fabric that does this out of the box yet, so you're assembling it yourself — which is normal. Here's a pattern that's worked well for me and strikes the right balance between thoroughness and overhead:

1. Validation notebook per layer — Runs as the final step in your pipeline after each layer loads. Checks row counts between source and target, null rates on key columns, primary key uniqueness, referential integrity between fact and dimension tables, and a handful of KPI aggregates (pick the 5–10 metrics your stakeholders care most about).
2. Write results to a `_data_quality_log` table in your gold layer — columns like `run_date`, `check_name`, `layer`, `expected_value`, `actual_value`, `pass_fail`. This gives you an audit trail for every run.
3. Fail the pipeline on critical checks — If a KPI is off by more than a defined threshold or row counts don't reconcile, raise an exception so the pipeline stops and alerts fire. Not every check needs to be a hard fail — categorize them as critical vs. warning.
4. Build a simple Power BI page on that log table — Now you and your stakeholders can see data quality trends over time, spot regressions early, and you have proof the numbers are right.

Regarding Great Expectations — it's a strong framework, but it adds operational complexity (managing the Python environment, expectation suites, data docs hosting). For a project where you're the primary engineer, the custom notebook approach gets you 80% of the value with far less overhead. You can always graduate to GE later if the project scales and more people are contributing.

4. General feedback

Your "make it work → make it correct → make it fast" approach is exactly right, and you're clearly aware of the technical debt on the backlog. A few things I'd add to your radar:

- Lineage documentation — Even a simple markdown file in your repo that maps "Gold table X is built from Silver tables Y and Z, joined on these keys, filtered by this logic." Future you will be grateful in three months when you've forgotten why something was built a certain way.
- Row-level security — If multiple clients share the gold layer, think about RLS early. Retrofitting it into a semantic model is painful. Centralizing into a single fact table with an RLS filter is usually cleaner long-term than tenant-specific fact tables.
- Semantic model as a contract — Once your KPIs are validated, treat the gold schema and semantic model as a contract. Changes should go through source control and be deliberate, not ad-hoc.
- Source system awareness — Understand what happens upstream. Does the Postgres source get bulk reloads? Are there late-arriving records? Soft deletes? These edge cases will bite you in production if your pipeline assumes clean, append-only data.

Your priority order should be: validate the numbers → add alerting and observability → set up deployment pipelines → then optimize (incremental gold, surrogate keys, RLS). Stakeholder trust is everything — if the dashboard numbers don't match on day one, you'll spend months recovering credibility.

Good luck — you're in a strong position.

If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!