Forum Discussion
Helper IIAfter Initial Deployment, How Can We Transfer Actual Data from Dev to Prod in Microsoft Fabric
I have implemented a Microsoft Fabric data pipeline with only two environments: Development (Dev) and Production (Prod).
In the Dev environment, data is loaded and maintained through incremental API ingestion, and historical data has been loaded initially using Excel files.
After the initial deployment, I understand that only metadata is promoted from Dev to Prod via the deployment pipeline. However, the actual data is not transferred between environments.
My question is: what is the recommended approach to handle the movement or synchronization of daily incremental data from Dev to Prod in this two-layer setup?
Rebuilding the pipeline or reloading data in Prod from scratch is not feasible due to time and effort constraints.
Is there a standard pattern in Fabric for handling this scenario (e.g., separate ingestion pipelines per environment, shared storage, or controlled data promotion strategies)?
Hi kasun_rumesh,
Not to my knoweldge. THe only known issue I see is related to warehouses and private link.
https://support.fabric.microsoft.com/known-issues/?product=Data%2520Factory&active=true&fixed=true&sort=published&issueId=1796
6 Replies
- tayloramy
Super User
Hi kasun_rumesh,
Generally, you do not want to move data between DEV and PRD.
In most set ups, DEV is connected to dev/tst systems, and PRD is connected to production systems, this is to avoid bogging down the production enviornment with development, and to comply with company privacy policies and local legsligation where applicable.
If you must use the same data for development and prod, what I would recommend is you load all your data to a "data" workspace that sits outside of dev/prd, and then use shortcuts to reference that data in dev and in prd.
This does carry the risk that something going wrong in dev can impact prod destructively, so be careful and ensure you have strong controls in dev.
kasun_rumeshYes, I agree with what you said. However, I am looking for a solution to transfer data between environments. I have also noticed that Lakehouse schemas are not carried over across environments.
- tayloramy
Hi kasun_rumesh,
There is no automated way to do that in Fabric. You will need to build pipelines or notebooks that move the data across.
- GilbertQ
Hi kasun_rumesh
There have been some great suggestions above. Another option you could do is to use the materialized lake views in order to transfer the data from your dev to prod environment. Here is the link. https://learn.microsoft.com/en-us/fabric/data-engineering/materialized-lake-views/overview-materialized-lake-view
