Forum Discussion
Incremental refresh data changes on dataflow
Dear all,
I have a Data flow with the below incremental refresh settings.
With this settings, will it detect and refresh data changes for the 2 years or only for the past 5 days?
I was assuming that it would detect data changes for the whole period and refresh no matter what the last 5 days.
Thanks in advance
Hey setis ,
unfortunately "detect data changes" does not work this way.
If you refresh today, the dataflow check for dates greater or equal 2020-06-23 (today - 5 days).
If a detect data changes column is configured Power BI stores the values of this column for the rows in the incremental refresh range. This can reduce the number of rows that will be refreshed.
This article explains this a little better: https://docs.microsoft.com/en-us/power-bi/transform-model/service-dataflows-incremental-refresh#:~:text=Using%20incremental%20refresh%20for%20Power,which%20incremental%20refresh%20can%20filter.
Regards,
Tom
Hi . setis
Incremental Refresh will make the part of the dataset to refresh much smaller, and as a result, the process would be much faster. If you have a modified DateTime (or updated DateTime) in your table,one better way to do that is to enable the option of "Detect Data Changes", and then choose the modified date or update date from the table. Notice that this is different from the OrderDate or transaction date. And not all tables have such a field.Then the process of incremental refresh can monitor that field, and only get rows that their date/time is after the latest date/time in that field in the previous refresh.
This video may better help you understanding the "detect data changes" in powerbi increment refresh.
https://www.youtube.com/watch?v=JsJWBr1_ktQ
In addition, you can keep an eye on this related thread.
Ps. "detect data changes" in the incremental refresh of dataflow should also be simlilar to that in the incremental refresh of dateset.
Best Regards,
Community Support Team _ Eason
2 Replies
Hey setis ,
unfortunately "detect data changes" does not work this way.
If you refresh today, the dataflow check for dates greater or equal 2020-06-23 (today - 5 days).
If a detect data changes column is configured Power BI stores the values of this column for the rows in the incremental refresh range. This can reduce the number of rows that will be refreshed.
This article explains this a little better: https://docs.microsoft.com/en-us/power-bi/transform-model/service-dataflows-incremental-refresh#:~:text=Using%20incremental%20refresh%20for%20Power,which%20incremental%20refresh%20can%20filter.
Regards,
Tom
Hi . setis
Incremental Refresh will make the part of the dataset to refresh much smaller, and as a result, the process would be much faster. If you have a modified DateTime (or updated DateTime) in your table,one better way to do that is to enable the option of "Detect Data Changes", and then choose the modified date or update date from the table. Notice that this is different from the OrderDate or transaction date. And not all tables have such a field.Then the process of incremental refresh can monitor that field, and only get rows that their date/time is after the latest date/time in that field in the previous refresh.
This video may better help you understanding the "detect data changes" in powerbi increment refresh.
https://www.youtube.com/watch?v=JsJWBr1_ktQ
In addition, you can keep an eye on this related thread.
Ps. "detect data changes" in the incremental refresh of dataflow should also be simlilar to that in the incremental refresh of dateset.
Best Regards,
Community Support Team _ Eason