Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Don't miss out! 2025 Microsoft Fabric Community Conference, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount. Prices go up February 11th. Register now.

Reply
Anonymous
Not applicable

Power BI Dataflow - Incremental refresh of entities and data flow schedule refresh

Hi All,

 

I have picked up the data flows in recent past and loving its features . I was able to create a data flow using Blank query and Copy+pasting Power BI from PBI desktop ( SQL as a data source) . Everything worked well so far. However, I got stuck when the Incremental refresh of Entities and Schedule refresh of Data flow came into picture. 

 

The data i pulled into this data flow contains at least 150 MB of data (approx. 70 lakhs of records in SQL terms) . I do not know which option to use for continuos refresh of this data as many of these records gets updated on hourly basis . Could some one explain me the differences between Incremental refresh of entities and Schedule refresh of data flows and suggest the best way of using data flows ?

 

Looking forward for the help !! 

 

Thanks,

G Venkatesh

8 REPLIES 8
v-lionel-msft
Community Support
Community Support

Hi @Anonymous ,

 

Schedule refresh:

By default, data are fully refreshed. That means that every time you refresh the dataset, all of the rows are replaced with the data in the underlying data source. While this kind of refresh ensures you have an exact copy of what is in the underlying data source, it can sometimes take a long time and be expensive on the database depending on how big the data is.

Incremental refresh:

Rather than refreshing the entire data source , you can set it up to only add the rows that are new. For example, you may have a data source that is updated daily with new sales transactions. Rather than rebuild the entire data source each day, you can just add the new transactions that occurred that day.

Summary: Schedule refresh to refresh all, incremental refresh can choose to refresh data for a certain period of time.

 

It's better to understand incremental refresh and schedule refresh (full refresh) by official document.
Using incremental refresh with Power BI dataflows 

Incremental refresh in Power BI Premium 

Configure scheduled refresh 

 

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Anonymous
Not applicable

Hi Lionel, 

 

Thanks for replying to this email. I got a clear picture about the different options of refreshing the data. Though these refreshes have different functionalities , similar names are always a confusion .

 

I have created few data flows in workspace and connected them to Azure datalake storage (default) . I work on service management data like Incidents , Work orders and Changes . we see plenty of new records being added every hour. At the same time, old records in the tables are also updated once the resource picks those tasks. Looks like i should utilize both the options (Schedule refresh as well as incremental refresh) to keep the data in report upto date. 

 

Any suggestions are welcomed !!!

 

Thanks,

G Venkatesh

Anonymous
Not applicable

Hello Team,

 

I want to circle back to the previous questions i have asked about Incremental refresh/Schedule refresh in relation with Dataflows . 

My data flow has lot of entities (say 25) now and i observed that the schedule refresh of data flows takes about 3 hours to refresh in peak hours of business. My users and colleagues within my team are concerned about data we are showing in reports. Could some one please assist me on how to tackle this situation ? 

 

To explain the issue again, I work on Service management data(Remedy ticketing tool) where we see new incidents coming in almost every minute . Some one suggested me to enable incremental refresh on this data. Also, I have old records getting updated with new time . Approximately few entities holds 50 lakhs of records and 3 hours long refresh doesnt hold good with clients. Do we have a solution to immediately implement ? 

 

Please provide me suggestions on tackling this situation ?

Anonymous
Not applicable

Does any one have anything to say about Incremental refresh ? 

 

I do not find any particular document to go through about the incremental refresh / schedule(full refresh) for dataflows . 

 

Kindly help . 

Hi @Anonymous ,

 

Based on your description, there is no doubt that you need to configure incremental refresh.

Incremental refresh is now generally available in the lastest version.

h10.PNG

https://docs.microsoft.com/en-us/power-bi/service-premium-incremental-refresh 

 

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Anonymous
Not applicable

Hi Lionel,

 

Thanks for the reply. I just read that in February 2020 updates. I will definitely going to set up Incremental refresh. 

 

However, One last question before I accept this as solution . Is it going to help me with both scenarios i have asked for. I have listed them again below. 

 

-Scenario 1 :   New record added to database.

 

Yes, I understand incremental refresh is going to add that to dataflow's dataset as well. 

 

Scenario 2 : Old record had some change . Will it be updated to dataflow's dataset using Incremental refresh ??

 

Kindly Assist !! 

 

 

Hi @Anonymous ,

 

Sorry, reply too late, has your problem been solved?

 

-Scenario 1 :   New record added to database.

Are you sure you mean "add to the database"? If it is a database, I can't answer.

 

"Scenario 2 : Old record had some change . Will it be updated to dataflow's dataset using Incremental refresh ??"

As long as the column names were not changed, and your old records can be overwritten by incremental refresh (Between RangeStart and RangeEnd), They can be refreshed.

 

Regarding scheduled refresh and incremental refresh, I made the following summary:

 Scheduled refreshIncremental refresh
Refresh timeDetermined by the time you configureWhen data changes
Refresh contentALLData that has changed in the last N days in the RangeStart and RangeEnd range
How to chooseThe data set is not very large and the data is updated infrequentlyotherwise

 

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Anonymous
Not applicable

Hey Lionel,

 

Thanks for the reply.  My first question would have confused you . But what i actually want to convey is 

 

- My Data source for dataflow  is relational database SQL in form of power query . I basically want to know if a new record is added to the database . How would i see that record getting added to the data flow as well?  Will Incremental refresh helps to add it to Data flow ? or Do i need to refresh my entire data flow ?

 

Thanks,

 G Venkatesh 

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Prices go up Feb. 11th.

Jan25PBI_Carousel

Power BI Monthly Update - January 2025

Check out the January 2025 Power BI update to learn about new features in Reporting, Modeling, and Data Connectivity.

Jan NL Carousel

Fabric Community Update - January 2025

Find out what's new and trending in the Fabric community.