Alternatives to Azure SQL for Metadata Database in Ingestion Framework Using Fabric Artifacts

[storks]

Hi everyone,

I'm working on a customer project where we’re building a generic ingestion and processing framework. Normally, I’d use an Azure SQL database to store metadata (e.g., data source info, transformations, load processes). However, in this case, we’re restricted from using Azure resources like Azure SQL, so all processes and storage must rely solely on Microsoft Fabric artifacts - like here.

I tried using a Fabric Warehouse as a metadata store, but I ran into performance issues, especially when many concurrent insert transactions were involved. Stability dropped significantly under high parallel write loads eg. logging.

Does anyone have recommendations for alternative Fabric artifacts for efficient metadata storage? Are there best practices for optimizing performance with parallel write operations in Fabric?

Thanks for any insights!

[shaunwatsonn11]

Thanks for the best advice

[storks · Accepted solution]

Fabric SQL Databases are the solution. Glad MSFT added this feature. 

[storks]

Hi @Anonymous ,

Thank you for the suggestion! Using a Lakehouse in Fabric is indeed an option I had considered. However, I’m specifically looking for a solution that supports parallel or concurrent inserts, as well as logging capabilities to capture metadata during Data Factory pipeline executions. This is something Azure SQL typically handles well as shown in the article https://techcommunity.microsoft.com/blog/fasttrackforazureblog/metadata-driven-pipelines-for-microsoft-fabric/3891651

Have you or anyone in the community had experience implementing a similar setup within Fabric? I’d love to hear any insights on optimizing for high-concurrency metadata storage and logging in this context.

[Anonymous]

Hi @storks 

Maybe you can consider using the Lakehouse in Fabric as an alternative. Not sure what's the structure and format of your meta data to be stored? In a lakehouse, you can store the files as they are under "Files\" in a lakehouse. If you want to store them in a table format, you can store them as delta tables under "Tables\". 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!