Forum Discussion
Issue with Incremental Refresh in Dataflows - Data from Previous Days Getting Deleted
Hello everyone,
I have set up two dataflows to create an 8-day data history. The first dataflow (dataflow1) stores data from the past 24 hours and refreshes at 10:30:00 PM, 12:00:00 AM, and 6:00:00 AM.
Next, I created a second dataflow (dataflow2) that is linked to dataflow1, and I configured an incremental refresh to retain data from the last 8 days by refreshing data from one day ago. I used a DateTime column for this purpose. Dataflow2 refreshes at 12:30:00 AM, 4:00:00 AM, and 6:00:00 AM.
However, I noticed that after 12:00:00 AM each day, the data from the previous day gets deleted in dataflow2, which is not the expected behavior for maintaining an 8-day data history.
I would like to understand what might be causing this data deletion and how to correct this configuration to preserve the complete 8-day history without data loss.
Thank you in advance for your help!
3 Replies
I believe the problem lies with the fact that Dataflow2 references a table in Dataflow1 that stores data for only 24 hours for the incremental refresh process. You may want to look at this article for more explanation and the solution - https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-create (Creating a dataflow by using Linked tables session). Please let me know if this sheds a bit of light on the solution.
Adedotun1, thank you for your response. Logically, it is not preferable to perform an incremental refresh on a dataflow that is linked to another dataflow, but I don't think that's the origin of my problem. Perhaps it's in the way I programmed my refresh.
Could be ... kindly let me (us) know when you find a solution to the problem
