Forum Discussion
Fabric how to manage data Dev/Test/Prod
Hello Perez0s0
Your colleague’s suggestion to reuse Production bronze/silver/gold data in lower environments is suitable provided it is implemented as read-only, fully parameterised, and supported by appropriate masking and guardrails.
However, it is advisable not to rely solely on this approach.
Your proposal for on-demand runs and subset data in Development and Test is an excellent default. By combining this with selective Production shortcuts for reference tables and occasional large-volume tests, you can achieve an effective balance between cost, speed, and quality.
In your Dev/Test workspaces, make sure shortcuts to Production bronze/silver/gold data are utilised. Transformations, such as notebooks and pipelines, are deployed and executed in Dev/Test, but the data itself is referenced from Production storage. This method minimises cost and avoids duplication of storage or compute for ingestion, providing realistic data for debugging and DAX/SQL logic. It is particularly advantageous in scenarios where cost pressures are significant and ingestion is resource-intensive. Potential PII risks can be mitigated through masking views, column-level security, test service principals, and robust data policies.
Hello Perez0s0
Your colleague’s suggestion to reuse Production bronze/silver/gold data in lower environments is suitable provided it is implemented as read-only, fully parameterised, and supported by appropriate masking and guardrails.
However, it is advisable not to rely solely on this approach.
Your proposal for on-demand runs and subset data in Development and Test is an excellent default. By combining this with selective Production shortcuts for reference tables and occasional large-volume tests, you can achieve an effective balance between cost, speed, and quality.
In your Dev/Test workspaces, make sure shortcuts to Production bronze/silver/gold data are utilised. Transformations, such as notebooks and pipelines, are deployed and executed in Dev/Test, but the data itself is referenced from Production storage. This method minimises cost and avoids duplication of storage or compute for ingestion, providing realistic data for debugging and DAX/SQL logic. It is particularly advantageous in scenarios where cost pressures are significant and ingestion is resource-intensive. Potential PII risks can be mitigated through masking views, column-level security, test service principals, and robust data policies.