Metadata driven pipeline

[DiKi-I]

Hi,

Can some one please share the metdata driven approach in Fabric with an example. I need to prepare a demo for the client.

[tayloramy · Accepted solution]

Hi @DiKi-I, 

Can you be more specific about what you need? 

I have a pipeline that runs a notebook, and the behaviour changes based on parameters sent to the notebook, and values in a metadata configuration lakehouse. 

I use this as my job orchestrator for Fabric - the notebook accepts a layer parameter, and then starts all the jobs on that layer ensuring that dependencies are met. 
My pipeline then runs this notebook 3 times, once for bronze, once for silver, and once for gold. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[pallavi_r · Accepted solution]

Hi @DiKi-I ,

There are multiple vlogs and youtube tutorial to build meta data driven architecture. Here you need to build  a generic pipeline and notebook and a metadata table which you can either build in fabric warehouse or sql db to store all the source and target connection details, the source and target layers, watermark columns.

https://blog.fabric.microsoft.com/en/blog/playbook-for-metadata-driven-lakehouse-implementation-in-m...

https://blog.fabric.microsoft.com/en-us/blog/create-metadata-driven-data-pipelines-in-microsoft-fabr...

Thanks,

Pallavi

[v-echaithra · Accepted solution]

Hi @DiKi-I ,

We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

Best Regards,
Chaithra E.

[v-echaithra]

Hi @DiKi-I ,

Thank you @pallavi_r  for your inputs.

I hope the information provided is helpful. I wanted to check whether you were able to resolve the issue with the provided solutions. Please let us know if you need any further assistance.

Thank you.

[pallavi_r · Accepted solution]

Hi @DiKi-I ,

There are multiple vlogs and youtube tutorial to build meta data driven architecture. Here you need to build  a generic pipeline and notebook and a metadata table which you can either build in fabric warehouse or sql db to store all the source and target connection details, the source and target layers, watermark columns.

https://blog.fabric.microsoft.com/en/blog/playbook-for-metadata-driven-lakehouse-implementation-in-m...

https://blog.fabric.microsoft.com/en-us/blog/create-metadata-driven-data-pipelines-in-microsoft-fabr...

Thanks,

Pallavi

[v-echaithra]

Hi @DiKi-I ,

May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.

Thank you.

[v-echaithra]

Hi @DiKi-I ,

Thank you for reaching out to Microsoft Community.

Please refer to this blog: Metadata Driven Pipelines for Dynamic Full and Incremental Processing in Azure SQL | Microsoft Commu...

[DiKi-I]

Thanks but I need an example for Fabric

[KevinChant]

Here's one practical example for you that is available in GitHub, so you can download and test it for yourself:

GitHub - bennyaustin/fabric-accelerator: Accelerator to build a Microsoft Fabric modern data platfor... 

[v-echaithra]

Hi @DiKi-I ,

Please refer to this document which has example too.
Metadata-Driven Pipelines in Microsoft Fabric: Automating Data Ingestion at Scale | by Yogitamatole ...

[tayloramy · Accepted solution]

Hi @DiKi-I, 

Can you be more specific about what you need? 

I have a pipeline that runs a notebook, and the behaviour changes based on parameters sent to the notebook, and values in a metadata configuration lakehouse. 

I use this as my job orchestrator for Fabric - the notebook accepts a layer parameter, and then starts all the jobs on that layer ensuring that dependencies are met. 
My pipeline then runs this notebook 3 times, once for bronze, once for silver, and once for gold. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.