After Initial Deployment, How Can We Transfer Actual Data from Dev to Prod in Microsoft Fabric

[kasun_rumesh]

I have implemented a Microsoft Fabric data pipeline with only two environments: Development (Dev) and Production (Prod).

In the Dev environment, data is loaded and maintained through incremental API ingestion, and historical data has been loaded initially using Excel files.

After the initial deployment, I understand that only metadata is promoted from Dev to Prod via the deployment pipeline. However, the actual data is not transferred between environments.

My question is: what is the recommended approach to handle the movement or synchronization of daily incremental data from Dev to Prod in this two-layer setup?

Rebuilding the pipeline or reloading data in Prod from scratch is not feasible due to time and effort constraints.

Is there a standard pattern in Fabric for handling this scenario (e.g., separate ingestion pipelines per environment, shared storage, or controlled data promotion strategies)?

[GilbertQ]

Hi @kasun_rumesh 

There have been some great suggestions above. Another option you could do is to use the materialized lake views in order to transfer the data from your dev to prod environment. Here is the link. https://learn.microsoft.com/en-us/fabric/data-engineering/materialized-lake-views/overview-materiali... 

[tayloramy]

Hi @kasun_rumesh,

Generally, you do not want to move data between DEV and PRD. 

In most set ups, DEV is connected to dev/tst systems, and PRD is connected to production systems, this is to avoid bogging down the production enviornment with development, and to comply with company privacy policies and local legsligation where applicable. 

If you must use the same data for development and prod, what I would recommend is you load all your data to a "data" workspace that sits outside of dev/prd, and then use shortcuts to reference that data in dev and in prd. 

This does carry the risk that something going wrong in dev can impact prod destructively, so be careful and ensure you have strong controls in dev. 

[kasun_rumesh]

Yes, I agree with what you said. However, I am looking for a solution to transfer data between environments. I have also noticed that Lakehouse schemas are not carried over across environments.

[tayloramy]

Hi @kasun_rumesh, 

There is no automated way to do that in Fabric. You will need to build pipelines or notebooks that move the data across.  

[kasun_rumesh]

Are there any known issues when using the Copy Job activity to transfer data from a Lakehouse?

[tayloramy · Accepted solution]

Hi @kasun_rumesh, 

Not to my knoweldge. THe only known issue I see is related to warehouses and private link. 
https://support.fabric.microsoft.com/known-issues/?product=Data%2520Factory&active=true&fixed=true&s...