topic Re: Issue with Incremental Refresh in Dataflows - Data from Previous Days Getting Deleted in Report Server

Issue with Incremental Refresh in Dataflows - Data from Previous Days Getting Deleted

Ibrahim10 — Mon, 24 Jul 2023 13:55:47 GMT

Hello everyone,

I have set up two dataflows to create an 8-day data history. The first dataflow (dataflow1) stores data from the past 24 hours and refreshes at 10:30:00 PM, 12:00:00 AM, and 6:00:00 AM.

Next, I created a second dataflow (dataflow2) that is linked to dataflow1, and I configured an incremental refresh to retain data from the last 8 days by refreshing data from one day ago. I used a DateTime column for this purpose. Dataflow2 refreshes at 12:30:00 AM, 4:00:00 AM, and 6:00:00 AM.

However, I noticed that after 12:00:00 AM each day, the data from the previous day gets deleted in dataflow2, which is not the expected behavior for maintaining an 8-day data history.

I would like to understand what might be causing this data deletion and how to correct this configuration to preserve the complete 8-day history without data loss.

Thank you in advance for your help!

Re: Issue with Incremental Refresh in Dataflows - Data from Previous Days Getting Deleted

AdedotunA — Mon, 24 Jul 2023 14:44:14 GMT

I believe the problem lies with the fact that Dataflow2 references a table in Dataflow1 that stores data for only 24 hours for the incremental refresh process. You may want to look at this article for more explanation and the solution - https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-create (Creating a dataflow by using Linked tables session). Please let me know if this sheds a bit of light on the solution.

Re: Issue with Incremental Refresh in Dataflows - Data from Previous Days Getting Deleted

Ibrahim10 — Mon, 24 Jul 2023 14:56:43 GMT

Adedotun1, thank you for your response. Logically, it is not preferable to perform an incremental refresh on a dataflow that is linked to another dataflow, but I don't think that's the origin of my problem. Perhaps it's in the way I programmed my refresh.

Re: Issue with Incremental Refresh in Dataflows - Data from Previous Days Getting Deleted

AdedotunA — Mon, 24 Jul 2023 15:36:22 GMT

Could be ... kindly let me (us) know when you find a solution to the problem