Forum Discussion
Deployment Pipeling Dataflows and linked entities
I have a solution though the use of Parameters to update the workspace to the different workspaces in the pipeline as a dataflows are progressed.
The issue was I need to combine Years worth of excel extracts from sharepoint for multiple data sources as I cannot access the servers directly due to lack of gateway.
Dataflow A - 24 excel files for Y data
Dataflow B - 24 Excel files for Z data
(there were more sources but simplifying it) These 2 dataflows, append and transform their data so they match field names and are appended to each other in dataflow C
in the dev space i only want to append 2 files for each data set and in the test and prod space i wanted all the data.
By Setting up a workspace parameter and changing the Navigation1 step to
Navigation{[workspaceName=""&Workspace&""]}[Data]This enables me to change the workspace the unloaded dataflow refreshes it data from as it moved through the pipeline.
Advocate IIII started playing around recently with dataflows and deployment pipelines.
From what I've experienced, the auto-bind feature of deployment pipelines won't work unless the linked entities are loaded.
I'm following the recommendation of having an extraction dataflow that simply loads the source tables, and a second dataflow connected (with linked tables) to the first in order to transform the tables.
If I don't load the linked tables on the second dataflow, they won't be auto-bound to their new workspace equivalents when deployed, and will keep referencing the entities in the original workspace (not what we normally intend). [Should this be an improvement request to allow auto-bind on not-loaded-linked-tables-from-other-dataflows?]
As for your comment regarding cascading refreshes, I think I read somewhere that the refreshes only affect the parts of the dataflow that depend on the triggering refresh. The cascading refreshes are not full-refresh. But I have not tested this so I'm not sure if that is correct.
I have a solution though the use of Parameters to update the workspace to the different workspaces in the pipeline as a dataflows are progressed.
The issue was I need to combine Years worth of excel extracts from sharepoint for multiple data sources as I cannot access the servers directly due to lack of gateway.
Dataflow A - 24 excel files for Y data
Dataflow B - 24 Excel files for Z data
(there were more sources but simplifying it) These 2 dataflows, append and transform their data so they match field names and are appended to each other in dataflow C
in the dev space i only want to append 2 files for each data set and in the test and prod space i wanted all the data.
By Setting up a workspace parameter and changing the Navigation1 step to
- agusmba
What I meant, trying to adapt it to your example, is that if your dataflow C is linking tables from dataflows A and B, but not loading them, when you deploy C on your TEST environment, the linked-but-not-loaded tables will point to A and B on the DEV environment. They are not auto-bound to A and B on the TEST environment unless the "load" tick is set.
I know, that was the problem I intially posted.
having them set to load would cause them all to fail as both dataflows A and B would try and refresh C at the same time.
- agusmba
Exactly, you'd need to set up different times for the refresh on A and B so that the cascading refreshes on C won't overlap.
If you don't load the linked tables, you can refresh A and B at the same time, and trigger C only once (manual/schedule). However in this last case you need to make sure that TEST-C is taking data from your TEST A and B instead of from DEV. You've probably already solved this with the use of parameters if I read you correctly, which is the way to go in this case.
In my case the dataflows in C were referenced via Id, so I'm not sure using a parameter for the workspace would be enough. Are you using the PowerPlatform dataflows connector or the legacy PowerBI one? For whatever reason I was reluctant to switch from the legacy to the new one.