Forum Discussion
Advocate IRefresh duration of a semantic model plugged on a dataflow
Dear all,
I would like to know if this situation is normal :
i am working in a premium capacity.
I load in a semantic model 2 tables from a dataflow Gen 1.
In the semantic model, there is no transformation in Power Query (the purpose is only to create visuals) no calculated Table / Columns and no Measures.
The size of the semantic model is about 1,6 Go. The 2 tables loaded have each 20 million records.
The refresh online of the semantic model take 1 hour. My feeling is that it is very long has there are absolutly no transformation, only the transfer of the data from the dataflow.
In advance thank you for your comments.
BrunoM Why the Refresh Might Be Slow
Here are some key considerations:
1. Dataflow Gen 1 Performance
- Gen 1 dataflows store data in CSV format on ADLS Gen2, which is less optimized for high-speed ingestion compared to Gen 2 (which supports Lakehouse, Warehouse, etc.).
- Reading large CSVs (20M rows per table) can be I/O intensive, especially if partitioning isn't used.
2. Semantic Model Import Mode
- Even without transformations, Power BI still loads data into VertiPaq memory engine, compresses it, and builds internal structures.
- A 1.6 GB model with 40M rows total can take time to process, especially if the dataflow refresh and semantic model refresh are not well synchronized.
3. No Incremental Refresh
- If you're doing a full refresh every time, Power BI re-ingests all 40M rows, which is costly.
- Consider enabling incremental refresh if your data supports it (e.g., has a date column).
4. Network and Capacity Load
- Even in Premium, performance can vary based on concurrent workloads, network latency, and backend throttling.
- Check the refresh history for signs of bottlenecks or retry attempts.
What You Can Do
- Switch to Dataflow Gen 2: It’s more performant and integrates better with Fabric’s Lakehouse architecture.
- Enable Incremental Refresh: This can drastically reduce refresh time if your data is time-partitioned.
- Use Data Pipelines: For better control over refresh sequencing and dependencies.
- Monitor Refresh Logs: Use refresh history and CSV logs to pinpoint slow phases (e.g., data read vs. model load).
2 Replies
- anilgavhane
Super User
BrunoM Why the Refresh Might Be Slow
Here are some key considerations:
1. Dataflow Gen 1 Performance
- Gen 1 dataflows store data in CSV format on ADLS Gen2, which is less optimized for high-speed ingestion compared to Gen 2 (which supports Lakehouse, Warehouse, etc.).
- Reading large CSVs (20M rows per table) can be I/O intensive, especially if partitioning isn't used.
2. Semantic Model Import Mode
- Even without transformations, Power BI still loads data into VertiPaq memory engine, compresses it, and builds internal structures.
- A 1.6 GB model with 40M rows total can take time to process, especially if the dataflow refresh and semantic model refresh are not well synchronized.
3. No Incremental Refresh
- If you're doing a full refresh every time, Power BI re-ingests all 40M rows, which is costly.
- Consider enabling incremental refresh if your data supports it (e.g., has a date column).
4. Network and Capacity Load
- Even in Premium, performance can vary based on concurrent workloads, network latency, and backend throttling.
- Check the refresh history for signs of bottlenecks or retry attempts.
What You Can Do
- Switch to Dataflow Gen 2: It’s more performant and integrates better with Fabric’s Lakehouse architecture.
- Enable Incremental Refresh: This can drastically reduce refresh time if your data is time-partitioned.
- Use Data Pipelines: For better control over refresh sequencing and dependencies.
- Monitor Refresh Logs: Use refresh history and CSV logs to pinpoint slow phases (e.g., data read vs. model load).
BrunoMThank you anilgavhane
I will try with a dataflow Gen 2 and also Pipeline (i do not know anything on this techno)