Forum Discussion

mcolb88's avatar
mcolb88
Helper III
5 years ago

Dataflow under the hood information

Does anyone have a link to information explaining what happens under the hood for dataflow processing?

 

Failures;

For example, if it fails, is the data intact from the last successful refresh?

If there are multiple entities, is it an all or nothing refresh, ie.e if any entity fails, then all entities rollback to the previous successful refresh?

 

Processing;

If there are multipe entities, other than obvious depedencies between them, are refreshes run in parallel? 

 

Performance;

Is there a way to get performance data on a dataflow, for example, a dataflow that refreshes daily, changes from a normal 10 minute duration to 20 minutes. HOw to determine which entity or data source id causing the issue?

 

Thanks

Dataflow
Need Help

1 Reply

  • jeffshieldsdev's avatar
    jeffshieldsdev
    Solution Sage
    5 years ago

    I don't have links but can say from experience, refreshes are all or nothing. Failure and all entities will be returned to their original state.

    Which successful or failed dataflows, you can look in Refresh History and download a .csv for each attempt and see when individual entities start and stop, and if Enhanced Compute Engine is being used.

    You should be able to determine from the log too if entities are being refreshed concurrently or consecutively.