Forum Discussion

storks's avatar
storks
Frequent Visitor
1 year ago
Solved

Alternatives to Azure SQL for Metadata Database in Ingestion Framework Using Fabric Artifacts

Hi everyone,

I'm working on a customer project where we’re building a generic ingestion and processing framework. Normally, I’d use an Azure SQL database to store metadata (e.g., data source info, transformations, load processes). However, in this case, we’re restricted from using Azure resources like Azure SQL, so all processes and storage must rely solely on Microsoft Fabric artifacts - like here.

I tried using a Fabric Warehouse as a metadata store, but I ran into performance issues, especially when many concurrent insert transactions were involved. Stability dropped significantly under high parallel write loads eg. logging.

Does anyone have recommendations for alternative Fabric artifacts for efficient metadata storage? Are there best practices for optimizing performance with parallel write operations in Fabric?

Thanks for any insights!

data engineering
Data Pipeline
Lakehouse
  • storks's avatar
    storks
    1 year ago

    Fabric SQL Databases are the solution. Glad MSFT added this feature. 

4 Replies

  • storks's avatar
    storks
    Frequent Visitor
    1 year ago

    Fabric SQL Databases are the solution. Glad MSFT added this feature. 

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi storks 

     

    Maybe you can consider using the Lakehouse in Fabric as an alternative. Not sure what's the structure and format of your meta data to be stored? In a lakehouse, you can store the files as they are under "Files\" in a lakehouse. If you want to store them in a table format, you can store them as delta tables under "Tables\". 

     

    Best Regards,
    Jing
    If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

  • storks's avatar
    storks
    Frequent Visitor
    1 year ago

    Hi Anonymous ,

    Thank you for the suggestion! Using a Lakehouse in Fabric is indeed an option I had considered. However, I’m specifically looking for a solution that supports parallel or concurrent inserts, as well as logging capabilities to capture metadata during Data Factory pipeline executions. This is something Azure SQL typically handles well as shown in the article https://techcommunity.microsoft.com/blog/fasttrackforazureblog/metadata-driven-pipelines-for-microsoft-fabric/3891651

    Have you or anyone in the community had experience implementing a similar setup within Fabric? I’d love to hear any insights on optimizing for high-concurrency metadata storage and logging in this context.