Forum Discussion
Fabric how to manage data Dev/Test/Prod
Hi,
we are in the middle of setting up our Fabric CI/CD environments. We are relatively new to this, so we could use some
advice.
We have set up three workspaces that serve as the core: 1. Dev, 2. Test, and 3. Prod.
Currently, we are discussing whether we should run our ingestion pipelines (bronze → silver → gold) in all three workspaces.
In our view, Production contains the complete production data and the full medallion structure.
For Test/Development, my colleague suggests using the bronze/silver/gold data from Production and pulling it into the development and test workspaces.
I disagree with this, even though I understand his concerns about cost and capacity, and why we would run all ingestion pipelines in each environment.
In my opinion, we can run the ingestion pipelines on-demand in Dev/Test, and potentially use a subset of the data, such as only the past year, instead of the full dataset.
Even though there are many valid ways to set this up depending on the situation, we are wondering whether these approaches are considered “no-gos,” or if they are acceptable and could work in practice.
Hello Perez0s0
Your colleague’s suggestion to reuse Production bronze/silver/gold data in lower environments is suitable provided it is implemented as read-only, fully parameterised, and supported by appropriate masking and guardrails.
However, it is advisable not to rely solely on this approach.
Your proposal for on-demand runs and subset data in Development and Test is an excellent default. By combining this with selective Production shortcuts for reference tables and occasional large-volume tests, you can achieve an effective balance between cost, speed, and quality.
In your Dev/Test workspaces, make sure shortcuts to Production bronze/silver/gold data are utilised. Transformations, such as notebooks and pipelines, are deployed and executed in Dev/Test, but the data itself is referenced from Production storage. This method minimises cost and avoids duplication of storage or compute for ingestion, providing realistic data for debugging and DAX/SQL logic. It is particularly advantageous in scenarios where cost pressures are significant and ingestion is resource-intensive. Potential PII risks can be mitigated through masking views, column-level security, test service principals, and robust data policies.
6 Replies
Hello Perez0s0
Your colleague’s suggestion to reuse Production bronze/silver/gold data in lower environments is suitable provided it is implemented as read-only, fully parameterised, and supported by appropriate masking and guardrails.
However, it is advisable not to rely solely on this approach.
Your proposal for on-demand runs and subset data in Development and Test is an excellent default. By combining this with selective Production shortcuts for reference tables and occasional large-volume tests, you can achieve an effective balance between cost, speed, and quality.
In your Dev/Test workspaces, make sure shortcuts to Production bronze/silver/gold data are utilised. Transformations, such as notebooks and pipelines, are deployed and executed in Dev/Test, but the data itself is referenced from Production storage. This method minimises cost and avoids duplication of storage or compute for ingestion, providing realistic data for debugging and DAX/SQL logic. It is particularly advantageous in scenarios where cost pressures are significant and ingestion is resource-intensive. Potential PII risks can be mitigated through masking views, column-level security, test service principals, and robust data policies.
Hi Perez0s0 ,
Option A – Full Isolation (Recommended)
Each workspace has:
Its own Bronze
Its own Silver
Its own Gold
Its own ingestion pipelines
Option B – Reuse Production Data
Dev/Test:
Pull bronze/silver/gold from Prod
Do not run ingestion pipelines independently
For Fabric, the common best-practice setup is:
Each Workspace Owns:
Lakehouse
Pipelines
Notebooks
Semantic models
And uses:
Parameterized connections
Environment-specific secrets
Deployment Pipelines for promotion
Also we can access the prod data using a shortcut for testing in lower environment if needed
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!Has anyone implemented automated schema validation between Dev → Test → Prod before deployment? Curious if people rely on pipeline tests or metadata comparison scripts
Well done
hello there. if each workspace has its own ingestion pipelines, how shoud deployment pipelines from dev->staging->prod be setup. do we bother deploying the bronze/silver/gold lakehouses?
Hi grdelgado
Thank you for your post. As this thread has been inactive for several months and was related to a different user's scenario, could you please create a new community post with the details of your issue? Please include the error message, screenshots (if applicable), the steps you've tried, and your environment details. This will help us investigate your specific scenario and provide the most accurate assistance.
Home - Microsoft Fabric Community
