Forum Discussion

tomkuria64's avatar
tomkuria64
Regular Visitor
2 months ago
Solved

Best practice for dynamic connection selection in Fabric Data Pipelines

Hi All,   I have a pipeline to extract data from an on-prem oracle DB and write to a Lakehouse on Fabric using Copy Data activity. The copy data activity requires connections to be setup to the sou...
data engineering
data pipeline
  • richbenmintz's avatar
    richbenmintz
    2 months ago

    I think in general you thinking is sound, I would only store the connection name in your config and use the Fabric API to get the connectionid as part of your pipeline, this will make the solution more portable as you move through environments without having to note and store all of the connection ids.

    As I am sure you know you will have to create seperate copy activities for each source type within your pipeline with a switch activity to route the operations or seperate pipeline for each type of data source you are copying data from.

richbenmintz's avatar
richbenmintz
Resident Rockstar
2 months ago

I think in general you thinking is sound, I would only store the connection name in your config and use the Fabric API to get the connectionid as part of your pipeline, this will make the solution more portable as you move through environments without having to note and store all of the connection ids.

As I am sure you know you will have to create seperate copy activities for each source type within your pipeline with a switch activity to route the operations or seperate pipeline for each type of data source you are copying data from.