Forum Discussion

AJAJ's avatar
AJAJ
Icon for Helper IV rankHelper IV
10 months ago
Solved

Copy Activity Dynamic Source Connection

Hi Currently my pipelines on dev and prod have the same source. So basically my ingestion pipeline + EDW pipeline run daily. Reads same source twice. By prod and then by Dev. In order to avoid doub...
data engineering
data pipeline
  • v-menakakota's avatar
    v-menakakota
    10 months ago

    Hi AJAJ ,

    I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you .

     

     

    Best Regards, 
    Community Support Team

v-menakakota's avatar
v-menakakota
Icon for Community Support rankCommunity Support
10 months ago

Hi AJAJ ,
Thanks for reaching out to the Microsoft fabric community forum. 

At the moment, Fabric pipelines don’t support dynamically switching the source system of a Copy activity at runtime based on a flag. While parameters exist, using them to change sources like this isn’t a supported or reliable pattern.

The real issue is that both Dev and Prod are reading from the source systems. In Fabric, the recommended approach is to read from the source only once, usually in Prod, land the data into Raw/Staging, and then let both Dev and Prod pipelines read from that staged data.

If Dev needs to test transformations, it should use the data already available in Raw/Staging (or a subset of it), instead of reading from the source again. This avoids double reads, reduces load on source systems, and keeps the setup simpler and more stable.

 

Best Regards, 
Community Support Team