Forum Discussion

VaibhavTiwari's avatar
VaibhavTiwari
Regular Visitor
23 days ago
Solved

Fabric SQL Database vs Warehouse

I am building a Metadata driven ETL Framework in Fabric and I am confused on where I should create my metadata storage and pipeline logs. As warehouse specializes in OLAP requirements, by definition ...
  • arabalca's avatar
    21 days ago

    Hi VaibhavTiwari​ ,

    As my colleague mentioned tayloramy​ , I would also go with a SQL Database, mainly because this kind of metadata-driven framework behaves much more like an OLTP workload.

    A Warehouse can work, and in fact I used one for similar application scenarios when SQL Database in Fabric was not GA yet. In the end, I migrated it to Database.

    The main issue is that when you start having many repetitive processes, writes, status updates, logs, etc., the CU consumption in the Warehouse becomes noticeably higher. You can optimize queries and processes, but there comes a point where, with enough frequency and concurrency, capacity usage still increases because it is not the type of workload the Warehouse is primarily designed for.

    For this use case, I would go directly with SQL Database.

    If this helped, please consider giving it a Like. If it solved your issue, please mark it as the Accepted Solution to help others facing the same problem.

    Thanks