Forum Discussion

storks's avatar
storks
Frequent Visitor
1 year ago
Solved

Alternatives to Azure SQL for Metadata Database in Ingestion Framework Using Fabric Artifacts

Hi everyone, I'm working on a customer project where we’re building a generic ingestion and processing framework. Normally, I’d use an Azure SQL database to store metadata (e.g., data source info, t...
data engineering
data pipeline
lakehouse
  • storks's avatar
    storks
    1 year ago

    Fabric SQL Databases are the solution. Glad MSFT added this feature. 

storks's avatar
storks
Frequent Visitor
1 year ago

Hi Anonymous ,

Thank you for the suggestion! Using a Lakehouse in Fabric is indeed an option I had considered. However, I’m specifically looking for a solution that supports parallel or concurrent inserts, as well as logging capabilities to capture metadata during Data Factory pipeline executions. This is something Azure SQL typically handles well as shown in the article https://techcommunity.microsoft.com/blog/fasttrackforazureblog/metadata-driven-pipelines-for-microsoft-fabric/3891651

Have you or anyone in the community had experience implementing a similar setup within Fabric? I’d love to hear any insights on optimizing for high-concurrency metadata storage and logging in this context.