Incremental Refresh Error

[rdini8]

Hi everyone, Incremental refresh stops working when I changed the data type of another column.

These are my applied steps. I made no changes to the datetime column used for filtering. I have also tried filtering before and after changing the data type but new data is not coming when I refresh the dataset. Also refresh always complete successfully but new data is not coming. If I remove the Changed Type step, it is working fine and new data comes when I refreshed. But I need to change the data type of the column. What could be the problem and solution?

[Anonymous]

Hi，

Thanks for the solutions @davidding  and @rajendraongole1 provided, and i want to offer some more information for user to refet to.

hello @rdini8 , 

the reason for the problem:
If you change the data type after you have set up incremental refresh, the current data type may be different from the data type in the historical interval during incremental refresh.

and the data may not be detected for this reason, so the data will not be returned.

Solution:
You may  consider reconfiguring incremental refresh after changing the data type

Best Regards!

Yolo Zhu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[rdini8]

Hi, I have tried filtering both before and after datatype changes. It's not working for either.

[Anonymous]

Hi @rdini8 

Thanks for your quick reply, you may not understand what I mean, are you already setting incremental refresh, and then back to Power Query to add the step of modifying the data type, because you have set up incremental refresh, so the original type is retained in the historical partition, when you modify the data type, after the partition is refreshed, you can't find the new data type in the original data, so you need to set the incremental refresh again.

Best Regards!

Yolo Zhu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[rdini8]

I haven't tried that. Should I disable and enable again?

[Anonymous]

Hi @rdini8 

Yes, you can try it, reset the incremental refresh, then try to refresh it agian.

Best Regards!

Yolo Zhu

[rdini8]

Thanks will try it. I was adding data and refreshing the dataset in the service continously. Is there a time duration before I should refresh or can i refresh everytime I add data?

[Anonymous]

Hi @rdini8 

If you just add data and the data type does not change, you do not need to refresh it every time.

Best Regards!

Yolo Zhu

[rdini8]

Hi, After days of checking it seems that it refreshes only once for a specific day. For example, Let's say we refresh at 12 pm for 25th July, we get all the rows on and before 12pm of 25th July. Any new rows added after 12pm won't come up anymore for 25th July. If we refresh on 26th, rows of 26th July will be added but not the rows of 25th added after 12pm. This is the same for all dates. It seems I can refresh only once for a particular date, at preferable at night like 11.55pm so that all rows of that date will be added. Is this some sort of bug or is it the way it is design?

[davidding]

just to add to @rajendraongole1 's answer. some data type change may not break query folding. here is an old article that highlights some of the possible changes: Changing data types that do not break query-folding in Power Query / Power BI (brunner.bi)  

[rdini8]

I changed only one column from whole number to text. Also I checked at the end of the applied steps, the native query is still available.

[rajendraongole1]

Hi @rdini8 - Incremental refresh relies heavily on query folding. If the data type change step breaks query folding, incremental refresh might not work correctly.To verify query folding, right-click on each step in the Power Query Editor and check if "View Native Query" is available. If it is not available after the data type change step, query folding is broken.

Reference link:

https://radacad.com/all-you-need-to-know-about-the-incremental-refresh-in-power-bi-load-changes-only

How to Implement Incremental Refresh in Power BI | phData

Hope it helps

Did I answer your question? Mark my post as a solution! This will help others on the forum!
Appreciate your Kudos!!

[rdini8]

I have checked the native query is still available at the end of the applied steps. What else can be the problem?