Forum Discussion
Best practice for dynamic connection selection in Fabric Data Pipelines
Hi All,
I have a pipeline to extract data from an on-prem oracle DB and write to a Lakehouse on Fabric using Copy Data activity. The copy data activity requires connections to be setup to the souce and destination respectively.
Is there a best practise to have the pipeline select the connections dynamically so that I can reuse the same pipeline for extracting data from multiple data sources.
I am thinking of creating a config table that hosts the source system name and connection guid. The parent pipeline iterates for each source system and the child pipeline would then lookup the config table to get the connection guid. Is this sound ok, or is there a better or best practice in such scenario's.
Thanks,
Tom
I think in general you thinking is sound, I would only store the connection name in your config and use the Fabric API to get the connectionid as part of your pipeline, this will make the solution more portable as you move through environments without having to note and store all of the connection ids.
As I am sure you know you will have to create seperate copy activities for each source type within your pipeline with a switch activity to route the operations or seperate pipeline for each type of data source you are copying data from.
4 Replies
I think in general you thinking is sound, I would only store the connection name in your config and use the Fabric API to get the connectionid as part of your pipeline, this will make the solution more portable as you move through environments without having to note and store all of the connection ids.
As I am sure you know you will have to create seperate copy activities for each source type within your pipeline with a switch activity to route the operations or seperate pipeline for each type of data source you are copying data from.
Hi tomkuria64
I think what you're looking for is known as a meta driven. Data Framework. Here is a link where you can download the repository and test this out with all the hard work already done in terms of stitching it all together. https://github.com/edkreuk/FMD_FRAMEWORK
Hi tomkuria64
Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.
Hi tomkuria64
We wanted to follow up to check if you’ve had an opportunity to review the previous responses. If you require further assistance, please don’t hesitate to let us know.
