When a dataflow refresh fails, what happens to the data in the entities?

[S4CSurrey]

When a dataflow refresh fails, what happens to the data in the entities?

Is the data only partially refreshed, or does the new data not come in at all?

I ask because after a failed dataflow refresh, I look at my report, and the data shown is cut off. This is because on my API end, there has been a timeout, which leads to an error in the dataflow, which results in a failure.

How do I instead stop the new data from coming in entirely? I don't want partial data.

[Greg_Deckler]

@S4CSurrey , unless I am mistaken, I believe the order of operations is:

1. Drop all rows in table

2. Load rows

So, it is adding rows up until the point of failure. So, if it fails after 10 rows are imported, you would have 10 rows. If it fails after 100 rows are imported, you would have 100 rows.

[S4CSurrey]

Alright, that's what I thought might be happening!

So in that case, how would I work around this issue? Ideally, I'd like to maintain the data that was previously loaded in a working refresh, and only bring in what was updated in the new refresh? The issue is, if the new refresh errors out, it would have cut off potential new data.

[v-lionel-msft]

Hi @S4CSurrey ,

When you fix the refresh failure problem and refresh again, the data refreshed for the first time will be overwritten.
You may try to refresh again and then check the dataset.

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[S4CSurrey]

That's the issue, I can't tell the future, I don't know when and if the data refresh will ever fail again. Due to that, the new "failed" refresh will bring in partial or empty data, overwriting my dataset. How do I prevent this?

[v-lionel-msft]

Hi @S4CSurrey ,

If you receive a notification that the refresh failed, maybe you can refresh it again manually.

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[S4CSurrey]

The data refreshes automatically 20 times a day. It's not feasible to manually intervene and run a manual refresh on every automated refresh.

There has to be some better error handling options in Dataflow!

[v-lionel-msft]

Hi @S4CSurrey ,

This is a good idea, please submit your idea .

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.