Forum Discussion

KimBeck's avatar
KimBeck
Regular Visitor
2 years ago
Solved

Best practice for using meta/config "table" for controlling loads?

For years I have been using a e.g. meta.LoadConfig table for controlling incremental/fullload and other options for controlling data load from sources to the classic Staging->DWH architecture. This h...
pipelines
  • KimBeck's avatar
    KimBeck
    2 years ago

    I found a way to work with a meta table in my Warehouse in my warehouse Workspace from a Bronze Workspace.

     

    I managed to do a lookup to the table by  using "Workspace" (not "External"), selecting Warehouse and then specifying the WarehouseID from the URL of my Workspace containing my Warehouse. Then I had to add the SQL Endpoint parameter also. Then I could access the config.meta table accross Workspaces.

     

     

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi KimBeck 

 

Thanks for using Microsoft Fabric Community.

As I understand that you've been successfully using a meta.LoadConfig table to manage data load control in your classic Staging->DWH architecture. That's a great approach!

For similar control in Microsoft Fabric, I found these relevant resources that discuss 'Metadata Driven Pipelines' which might help you.

Link1 : Metadata Driven Pipelines for Microsoft Fabric and

Link2 : Metadata Driven Pipelines for Microsoft Fabric – Part 2, Data Warehouse Style.

These articles focus on using metadata for data pipelines within Fabric, which aligns well with your existing approach.

 

I hope this information helps. Please do let us know if you have any further questions.

 

Thanks.