Forum Discussion
Dataflow chains
I set up a series of dataflows and set refresh for the 3 upstream.
This set of dataflows siphon pretty much a couple millions records and I have only 1 hour time to make its data fully flow downstream. I read some that if A > C and B > C, whereas B is lesser stream, its linked entity could be disable in C so that when B is refresh, C would not refresh in linked yet follow A schedule.
But when I tried that, my C dataflow got error message as following:
Error: On-Prem execution not supported for entity '2d20a598-b3b3-4ffe-9882-6e8db6ba1c99/3e9ec4a0-55ed-45b7-b2e0-64d3e98914e2/ItemProdCls_Active' since it refers to another entity.. RootActivityId = d172af1b-bb8f-4168-b384-be0469fce80f.Param1 = 2d20a598-b3b3-4ffe-9882-6e8db6ba1c99/3e9ec4a0-55ed-45b7-b2e0-64d3e98914e2/ItemProdCls_Active Request ID: d9e73938-69ad-ad3f-b44c-c3e15a65ab60.
When B linked entity is enable load again, things get back to normal.
Since refresh schedule is not always on time, even though I set B refresh 30 minute later than A, sometimes I noticed both A and B in refresh progress at the same time. My question for this is how will C is processed?
Would you have any other suggestion rather than stagering time of refresh schedule?
Thanks for your advise!
Best regards.
Hi rdnguyen ,
Is dataflow C built on dataflow A and dataflow B? If you would like to know the refresh logic of linked entities, you may refer to Link entities between dataflows - Power Query | Microsoft Docs. For links between workspaces and Links in the same workspace, the refresh logics are different.
- Refresh for links from entities in different workspaces behaves like a link to an external data source. When the dataflow is refreshed, it takes the latest data for the entity from the source dataflow. If the source dataflow is refreshed, it doesn't automatically affect the data in the destination dataflow.
- When data refresh occurs for a source dataflow, that event automatically triggers a refresh process for dependent entities in all destination dataflows in the same workspace, including any calculated entities based on them. All other entities in the destination dataflow are refreshed according to the dataflow schedule. Entities that depend on more than one source refresh their data whenever any of their sources are refreshed successfully.
- The entire refresh process is committed at once. Because of this, if the data refresh for the destination dataflow fails, the data refresh for the source dataflow fails as well.
Best Regards,
Community Support Team _ Caiyun
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly. If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!
2 Replies
Hi rdnguyen ,
I have run into this same problem in the past. There is no "timer" function native to Power BI that would let you daisy-chain the dataflows. Therefore, the way you are doing it is probably the best way. It just takes experimentation to get the flow and timing right.
I have created an idea so please vote for this (and have yoru friends and family vote too) and maybe this will get on Microsoft's radar.
https://ideas.powerbi.com/ideas/idea/?ideaid=e86653d6-adab-ec11-826d-501ac5248c99
Hi rdnguyen ,
Is dataflow C built on dataflow A and dataflow B? If you would like to know the refresh logic of linked entities, you may refer to Link entities between dataflows - Power Query | Microsoft Docs. For links between workspaces and Links in the same workspace, the refresh logics are different.
- Refresh for links from entities in different workspaces behaves like a link to an external data source. When the dataflow is refreshed, it takes the latest data for the entity from the source dataflow. If the source dataflow is refreshed, it doesn't automatically affect the data in the destination dataflow.
- When data refresh occurs for a source dataflow, that event automatically triggers a refresh process for dependent entities in all destination dataflows in the same workspace, including any calculated entities based on them. All other entities in the destination dataflow are refreshed according to the dataflow schedule. Entities that depend on more than one source refresh their data whenever any of their sources are refreshed successfully.
- The entire refresh process is committed at once. Because of this, if the data refresh for the destination dataflow fails, the data refresh for the source dataflow fails as well.
Best Regards,
Community Support Team _ Caiyun
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly. If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!
