Need Help in Building a Metadata Driven Pipeline using Fabric Notebooks (Code Based only)

Hi Anonymous ,

1.Design a metadata/config table with fields like:

• source_file_pattern : e.g., BB_End_user_\d{6}
• target_table: e.g., Bronze.BB_End_user
• load_type: full or incremental
• date_key, start_date,last_loaded_date
• status, rows_inserted, rows_updated

This enables dynamic pipeline orchestration and tracking

2. Use the metadata table to determine load type:

• Full Load: Drop and recreate the Delta table or overwrite.
• Incremental Load: Filter based on date_key or last_loaded_date

In Fabric Notebooks, you can trigger notebook execution via:

• Interactive run
• Pipeline activity
• Scheduler plan

3. Notebook Execution and Orchestration

Use Fabric Pipelines to orchestrate notebook execution:

• Lookup activity to fetch config rows.
• For each row, pass parameters to a child notebook.
• Use Run as pipeline activity for scalable execution

Refer- Metadata Driven Pipelines for Microsoft Fabric

Hope this helps!