Forum Discussion
Automating dynamic data source changes in Microsoft Fabric with Dataflows Gen2 and pipelines
Hi TesnimeBS_,
Have you considered creating a single sementic model for your Power BI report and apply Row Level Security (RLS) on that report: https://learn.microsoft.com/en-us/fabric/security/service-admin-row-level-security? You could then write all your data to the same model and mitigate voluminous semantic schemas.
You still need to create separate connections to the servers, but you could use a loop in the data pipeline to select each of the servers automatically. Then, your data pipeline remains managable and clear.
Kind regards,
Fabian
Hi TesnimeBS_,
You could for example create a CSV file within your Lakehouse that holds all the Fabric connection ID's that you created for your different servers. With a Lookup within the data pipeline, you could get the rows from the CSV a use that in the ForEach activity to get the Fabric connection ID's per row. Then you could use the Dynamic Content option within the copy activity to use that ID as the connection. An example is given here:
https://community.fabric.microsoft.com/t5/Data-Pipelines/Dynamically-set-Copy-data-Destination-Connection-or-Notebook/m-p/4059192/highlight/true#M4703
HI FabianSchut
Thank you for your response!
Could you please guide me on the following points:
How to configure a Lookup activity to dynamically retrieve server information?
How can I create a loop in a data pipeline to automatically switch between servers?
How to pass this dynamic information (server name, database, etc.) into a data copy or extraction step?
If you have any concrete examples or links to tutorials, that would be greatly appreciated!
Thank you in advance for your help!
Kind regards,
Tesnime
Hi TesnimeBS_,
You could for example create a CSV file within your Lakehouse that holds all the Fabric connection ID's that you created for your different servers. With a Lookup within the data pipeline, you could get the rows from the CSV a use that in the ForEach activity to get the Fabric connection ID's per row. Then you could use the Dynamic Content option within the copy activity to use that ID as the connection. An example is given here:
https://community.fabric.microsoft.com/t5/Data-Pipelines/Dynamically-set-Copy-data-Destination-Connection-or-Notebook/m-p/4059192/highlight/true#M4703