Expert opinion/tips regarding incremental refresh with Semantic models/Dataflow

[AutoJL]

Hi fellow community!

I need some expert advice on the topic above mentioned.

To start off since this topic is related to both semantic models and Dataflows I originally created the same topic on the "Desktop" forums. I was advised there to also post here since there might more extra knowledge to gain regarding Dataflows.

Let me put some context in order to better explain the situation.

I have a powerbi report that is built from plain txt files (actually .log files that contain a mixture of plain text as well as flattened jsons). I am storing this files in either Sharepoint/Azure blob storage. In order to enhance this report I want to make use of the incremental refresh feature since this .log files are pretty heavy: 100k+ lines each file with multiple files per day and per system to analyze and multplie systems to store info and extrac insights from. As you can imagine all the data transformations get very heavy, specially as multiple days worth of information are stored ( and since its log information past files are never modified).

I have been searching some forums/blogs online to fully get grasp on how incremental refresh works (and of course the MS Learn documentation) , specially considering that my datasources cant fold querys to make this easier. As it happens theres some information that this is still something that can be overcome ( like this for example: Power BI incremental refresh with binary files).

I have tested with the Semantinc model Incremental refresh and it definetely looks like it works and im confident it will also work if using Dataflows.

My questions to you experts come here:

What should I should use?

Any specific recomendations on wether to use the Semantic model vs the Dataflows Incremental Refresh?

Any restrictions I need to take into account?

One thing to take into account is that I do have access to Premium Per User workspace/license in my organization.

Any help, tips, suggestions, related content to enhance knowledge on this topic and situations (Blogs/Youtube videos, etc) is more than welcome.

Many thanks!

[AutoJL]

Hi everyone!

Thanks @christinepayton , @Anonymous  and @v-kpoloju-msft  for stopping by and helping out, much appreciated!.

Since @v-kpoloju-msft basically combines all the information and adds some links also, I will mark that as a solution so people can find the more detailed post but kudos to everyone for the help!

Thanks everyone!

[v-kpoloju-msft · Accepted solution]

Hi @AutoJL,

Thank you for reaching out to the Microsoft Fabric Community Forum.

After reviewing the issue of tips regarding incremental refresh with Semantic models and dataflow, here are some tips regarding semantic model and dataflow.

Please go through the following tips for better understanding:

Semantic model:

• Incremental refresh in semantic models is highly efficient especially with large datasets. It enables refreshing only the new or updated data, rather than the entire dataset.
• Implementing incremental refresh in a semantic model requires setting up parameters for date and time and defining incremental refresh policies. This could be straight forward if your data includes time-based attributes.
• Semantic models provide enhanced flexibility for handling complex calculations, relationships, and DAX measures. If your report heavily relies on these features, semantic models may be the more suitable choice.
• Verify that your data source supports query folding to achieve optimal performance. If query folding is not supported by your current setup, the performance improvements may be limited.

Dataflow:

• Dataflows are highly effective for managing the ETL Extract, Transform, Load process. They enable the preparation and cleaning of data before it is loaded into your semantic model, which can enhance both performance and manageability.
• Dataflows are reusable across various reports and can be shared with different users, for consistency and collaboration.
• Incremental refresh in dataflows can significantly reduce the time and resources needed for data processing. However, it may not support the more complex transformations available in Power BI Desktop.
• Since you have a Premium Per User license, you can utilize features like enhanced dataflows, which provide improved performance and additional capabilities compared to standard dataflow.
• Use dataflows for initial data extraction and transformation to reduce the load on your semantic model and improve performance. Then, use the semantic model for complex calculations and visualizations.

if you Favor complexity and performance in analysis, go for Semantic models with incremental refresh parameters. If you want a reusable and scalable ETL layer, Dataflows might be the better choice.

Please go through the below documentation links for better understanding:

Understand and optimize dataflows refresh - Power BI | Microsoft Learn

Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn

Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

[christinepayton]

If you're querying files, Power BI really prefers combining a smaller number of large files vs a large number of small files, so if it's possible to automate combining the files before they get to Power BI that will be preferable (if speed is an issue). Multiple files per day will get out of control really quickly. 

Incremental refresh on dataflows is premium, but you have premium, so you should be good there. I tend to put queries in dataflows when the query will be reused in multiple models. 

The incremental settings for dataflows are at the dataflow level, so I'd probably put it in its own dataflow if you go that route unless you have other things using the same incremental settings. 

[Anonymous]

Either Dataflow or Semantic model will work in your case for incremental refresh. Use the Dataflow if more than one semantic model uses the same data. Use the semantic model incremental refresh if it is the only mode.