Forum Discussion
Advocate IVDataflows refresh for linked entities failing
Hi,
Does anyone know if it's possible to refresh dataflows in parallel (lets call those dataflows 'A'), when they are connected to another dataflow ('B') which contains linked entities from 'A'? I can only seem to have one dataflow refesh at a time, the other will fail. There is also a minimum 30 min interval between scheduling refreshes, which would take too long to schedule all the dataflows I need. There's currently 9 dataflows set to incremental refresh so they only take about 5 mins each, but would need to be staged over 4.5 hours which isn't acceptable for my case (I have about a 1 hour window available to have refreshes completed) so currently the only way is to do this manually every day :(.
Alternatively I'm also wondering if this is actually/partially consequent of other current recognised issues around scheduled refreshes failing, and that the issue might soon just go away with a fix coming?
Any help appreciated!
I originally used this architecture too.
As you've noticed, when your first A Level dataflow starts refreshing, it puts a lock on downstream dataflows in the same workspace...so if the second A Level dataflow tries to refresh, it won't be able to get a lock on the B Level dataflow and will fail.
My workaround has been to create another workspace--my "publish" workspace--that has your B Level dataflow with all the A Level entities contained as Linked Entities.
B won't automatically refresh with the A Level dataflows, but since all B contains is Linked Entities, a scheduled refresh should run quickly.
With the B Level dataflow removed from your inital workspace, your A Level dataflows will be able to refresh concurrently.
Also, you could look at using MS Flow/Power Automate or Logic Apps to orchestrate a more granular refresh schedule--you can trigger Power BI refresh from both 🙂
5 Replies
- jeffshieldsdev
Solution Sage
I originally used this architecture too.
As you've noticed, when your first A Level dataflow starts refreshing, it puts a lock on downstream dataflows in the same workspace...so if the second A Level dataflow tries to refresh, it won't be able to get a lock on the B Level dataflow and will fail.
My workaround has been to create another workspace--my "publish" workspace--that has your B Level dataflow with all the A Level entities contained as Linked Entities.
B won't automatically refresh with the A Level dataflows, but since all B contains is Linked Entities, a scheduled refresh should run quickly.
With the B Level dataflow removed from your inital workspace, your A Level dataflows will be able to refresh concurrently.
Also, you could look at using MS Flow/Power Automate or Logic Apps to orchestrate a more granular refresh schedule--you can trigger Power BI refresh from both 🙂
Thanks jeffshieldsdev ! I have the same exact issue too.
Anyway what should be the correct behaviour assuming all issues about scheduled refrehes will be fixed? Is the best practice to
1) create the "publish" workspace separating A and B level Dataflows OR
2) leave both A and B Level dataflows in the same workspaces (assuming the refresh will complete correclty)?
Thanks
- jeffshieldsdev
If you want your A dataflows to be able to refresh concurrently, your B level cannot be in the same workspace--so option 1.
Techincally it could be--if you disabled load on the linked entities in it--but that would make a copy of the A Level dataflow data, which is redundant.
- uberdube
Hi,
Does anyone know if it's possible to refresh dataflows in parallel (lets call those dataflows 'A'), when they are connected to another dataflow ('B') which contains linked entities from 'A'? I can only seem to have one dataflow refesh at a time, the other will fail. There is also a minimum 30 min interval between scheduling refreshes, which would take too long to schedule all the dataflows I need. There's currently 9 dataflows set to incremental refresh so they only take about 5 mins each, but would need to be staged over 4.5 hours which isn't acceptable for my case (I have about a 1 hour window available to have refreshes completed) so currently the only way is to do this manually every day :(.
Alternatively I'm also wondering if this is actually/partially consequent of other current recognised issues around scheduled refreshes failing, and that the issue might soon just go away with a fix coming?
Any help appreciated!
