Forum Discussion

Kuladeep's avatar
Kuladeep
Icon for Advocate II rankAdvocate II
3 months ago
Solved

What’s the best for Metadata driven Pipeline?

Hi Community,   I am trying to optimize our existing pipeline. A former colleague built our metadata-driven pipelines using a Fabric SQL Database.   However, we have been facing some issues with ...
data engineering
  • kr_richbenmintz's avatar
    kr_richbenmintz
    2 months ago

    So the pattern that I follow and tend to advocate is:

     

    • Yaml configuration files that store the config attributes and values
      • Benefits:
        • config and and values are all source controlled and tokenizable for environment specific releases
        • Release Pipelines deploys configurations to config lakehouse
        • Schema is easily evolved, new datasources and additional configuration attributes are easily created without database schema changes and management
        • No Dependency on additional Fabric or external SKU
      • Potential Con - Notebook or Fabric Function required to generate dictionary that is useable by ADF if data is being moved using copy command
    • Orchestration Notebooks and Executor Notebooks for data Orchestration and Execution
      • Orchestrator gets list of configs to run and executes through notebookUtils.notebook.runMultiple or bespoke parallel procesing implementation
    • All Pipeline and Process Metadata generated and potentially required by future/upstream/downstream processes stored in KQL database

    Hope that is a little Helpful

alicegray's avatar
alicegray
Icon for Helper I rankHelper I
3 months ago

Hey Kuladeep 

 Implementing a metadata-driven framework in Fabric requires configuring: 

  • Metadata Store 
  • Control (Orchestrator) Pipeline 
  • Generic Child Pipelines 
  • Watermark Management 
  • Error Handling & Retry Logic 

    These elements ensure smooth querying and updates. Would you like to know about specific implementation, such as the SQL schema for the control tables, a sample pipeline expression, or a notebook-based metadata approach? Please let me know! 

 

  • Kuladeep's avatar
    Kuladeep
    Icon for Advocate II rankAdvocate II
    3 months ago

    Hi Alicegray,
    We have all the set up that you mentioned (orchestration pipeline, child pipelines, watermark management, error handling and retry logic) and it has been working smooth except these unexpected hick ups in middle. We are using the SQL DB for Metadata Store at the very first step and it is giving up on us easily. There is nothing to retry except force restart of the capacity.