Metadata driven pipeline

Hi DiKi-I, 

Can you be more specific about what you need? 

I have a pipeline that runs a notebook, and the behaviour changes based on parameters sent to the notebook, and values in a metadata configuration lakehouse. 

I use this as my job orchestrator for Fabric - the notebook accepts a layer parameter, and then starts all the jobs on that layer ensuring that dependencies are met. 
My pipeline then runs this notebook 3 times, once for bronze, once for silver, and once for gold. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

Hi DiKi-I ,

There are multiple vlogs and youtube tutorial to build meta data driven architecture. Here you need to build  a generic pipeline and notebook and a metadata table which you can either build in fabric warehouse or sql db to store all the source and target connection details, the source and target layers, watermark columns.

https://blog.fabric.microsoft.com/en/blog/playbook-for-metadata-driven-lakehouse-implementation-in-microsoft-fabric/

https://blog.fabric.microsoft.com/en-us/blog/create-metadata-driven-data-pipelines-in-microsoft-fabric/

Thanks,

Pallavi

Hi DiKi-I ,

We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

Best Regards,
Chaithra E.