topic Re: Best practice for using meta/config "table" for controlling loads? in Pipelines

Best practice for using meta/config "table" for controlling loads?

KimBeck — Fri, 08 Mar 2024 10:30:35 GMT

For years I have been using a e.g. meta.LoadConfig table for controlling incremental/fullload and other options for controlling data load from sources to the classic Staging->DWH architecture. This has worked very well.

Is there a "wetted" best-practice process for similar control in Fabric Data Factory? -anyone using a meta/config file for controlling this?

- I want to try accomplish this without having to provision e.g. an Azure SQL Database for controlling this. 🙂

Re: Best practice for using meta/config "table" for controlling loads?

v-cboorla-msft — Sat, 09 Mar 2024 05:34:48 GMT

Hi KimBeck

Thanks for using Microsoft Fabric Community.

As I understand that you've been successfully using a meta.LoadConfig table to manage data load control in your classic Staging->DWH architecture. That's a great approach!

For similar control in Microsoft Fabric, I found these relevant resources that discuss 'Metadata Driven Pipelines' which might help you.

Link1 : Metadata Driven Pipelines for Microsoft Fabric and

Link2 : Metadata Driven Pipelines for Microsoft Fabric – Part 2, Data Warehouse Style.

These articles focus on using metadata for data pipelines within Fabric, which aligns well with your existing approach.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

Re: Best practice for using meta/config "table" for controlling loads?

KimBeck — Sat, 09 Mar 2024 10:29:49 GMT

Thanks for the input.

yes, I am looking for a way to port my existing dataflow control knowledge to Fabric conditions.

Right now, I want to build a Bronze LH -> Silver LH -> Gold WH architecture.

I think that currently, the best solution would be to have a "config" schema in the Gold WH, then I can update the meta/config/control tables in SSMS like I am used to.

The connect from Data Factory to this table and retrive the metadata.

The proposed solutions are interesting, they do however suggest provisioning a Azure SQL Database for the meta data.

Re: Best practice for using meta/config "table" for controlling loads?

v-cboorla-msft — Wed, 13 Mar 2024 11:32:07 GMT

Hi KimBeck

Following up on your query to see if you've reached a resolution. And was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks

Re: Best practice for using meta/config "table" for controlling loads?

KimBeck — Wed, 13 Mar 2024 13:10:14 GMT

I found a way to work with a meta table in my Warehouse in my warehouse Workspace from a Bronze Workspace.

I managed to do a lookup to the table by using "Workspace" (not "External"), selecting Warehouse and then specifying the WarehouseID from the URL of my Workspace containing my Warehouse. Then I had to add the SQL Endpoint parameter also. Then I could access the config.meta table accross Workspaces.