Forum Discussion
Intraday Data for Reporting
Hi Digidank ,
Thanks for using Fabric Community.
Synapse Data Warehouse (DW) can definitely handle your intraday data updates. It's designed for large-scale data ingestion and frequent updates, so capacity shouldn't be a major obstacle. Here's how we can tackle both nightly and intraday data:1. Nightly Data:
Keep using your existing Spark jobs to process and load nightly data files into Synapse DW. This is a familiar and efficient approach.
2. Intraday Updates:
Break down your intraday data into smaller batches and ingest them into Synapse DW throughout the day. This provides near real-time updates for your reports.
Important Note:
While Fabric notebooks are a valuable tool for data exploration and analysis, they currently cannot directly load data into the Data Warehouse. We recommend using Data Factory pipelines for this purpose.
Additional Tips:
Partition your tables in Synapse DW to optimize query performance for both historical and intraday data.
Consider using materialized views for frequently used aggregations on intraday data to further improve reporting speed.
Docs to refer -
Ingesting data into the warehouse - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Please do let me know incase of further queries.
Hi Digidank ,
Thanks for using Fabric Community.
Synapse Data Warehouse (DW) can definitely handle your intraday data updates. It's designed for large-scale data ingestion and frequent updates, so capacity shouldn't be a major obstacle. Here's how we can tackle both nightly and intraday data:
1. Nightly Data:
Keep using your existing Spark jobs to process and load nightly data files into Synapse DW. This is a familiar and efficient approach.
2. Intraday Updates:
Break down your intraday data into smaller batches and ingest them into Synapse DW throughout the day. This provides near real-time updates for your reports.
Important Note:
While Fabric notebooks are a valuable tool for data exploration and analysis, they currently cannot directly load data into the Data Warehouse. We recommend using Data Factory pipelines for this purpose.
Additional Tips:
Partition your tables in Synapse DW to optimize query performance for both historical and intraday data.
Consider using materialized views for frequently used aggregations on intraday data to further improve reporting speed.
Docs to refer -
Ingesting data into the warehouse - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Please do let me know incase of further queries.
