Forum Discussion
Deployment Pipeling Dataflows and linked entities
Im trying to utilise deployment pipelines however having an issue where a master dataflow (fed from other dataflows in the pipeline) isnt updating to the new workspaces as it is promoted unless the dataflow entities have enable load ticked.
This is a problem as there are 12 dataflows going into this master flow and I do not want 12 refreshes triggered on it.
Is there a way to disable the triggered refresh without disabling enable load?
I have a solution though the use of Parameters to update the workspace to the different workspaces in the pipeline as a dataflows are progressed.
The issue was I need to combine Years worth of excel extracts from sharepoint for multiple data sources as I cannot access the servers directly due to lack of gateway.
Dataflow A - 24 excel files for Y data
Dataflow B - 24 Excel files for Z data
(there were more sources but simplifying it) These 2 dataflows, append and transform their data so they match field names and are appended to each other in dataflow C
in the dev space i only want to append 2 files for each data set and in the test and prod space i wanted all the data.
By Setting up a workspace parameter and changing the Navigation1 step to
Navigation{[workspaceName=""&Workspace&""]}[Data]This enables me to change the workspace the unloaded dataflow refreshes it data from as it moved through the pipeline.
7 Replies
- lbendlin
Super User
You are supposed to use scaled down data sources in the DEV and ITG workspaces. Using the same source in all workspaces defeats the purpose somewhat. You can also choose to manually refresh the dataflow on demand.
Thats exactly what im trying to do via use of parameters to bring in less files from sharepoint for each of the 12 dataflows.
To utilise different sharepoint locations would be couter intuitive as this would be duplication the data storage, It would also not alleviate the problem of combining the datasources into a single entity.
Each Dataflow represents a different system providing eqivilent data, Each systems data needs to be transformed and cleansed so they can be appended together to create a dataset which can report across all systems to give a buisness wide view. There is no Direct database access possible
agusmbaAdvocate III
I started playing around recently with dataflows and deployment pipelines.
From what I've experienced, the auto-bind feature of deployment pipelines won't work unless the linked entities are loaded.
I'm following the recommendation of having an extraction dataflow that simply loads the source tables, and a second dataflow connected (with linked tables) to the first in order to transform the tables.
If I don't load the linked tables on the second dataflow, they won't be auto-bound to their new workspace equivalents when deployed, and will keep referencing the entities in the original workspace (not what we normally intend). [Should this be an improvement request to allow auto-bind on not-loaded-linked-tables-from-other-dataflows?]
As for your comment regarding cascading refreshes, I think I read somewhere that the refreshes only affect the parts of the dataflow that depend on the triggering refresh. The cascading refreshes are not full-refresh. But I have not tested this so I'm not sure if that is correct.
I have a solution though the use of Parameters to update the workspace to the different workspaces in the pipeline as a dataflows are progressed.
The issue was I need to combine Years worth of excel extracts from sharepoint for multiple data sources as I cannot access the servers directly due to lack of gateway.
Dataflow A - 24 excel files for Y data
Dataflow B - 24 Excel files for Z data
(there were more sources but simplifying it) These 2 dataflows, append and transform their data so they match field names and are appended to each other in dataflow C
in the dev space i only want to append 2 files for each data set and in the test and prod space i wanted all the data.
By Setting up a workspace parameter and changing the Navigation1 step to
Navigation{[workspaceName=""&Workspace&""]}[Data]This enables me to change the workspace the unloaded dataflow refreshes it data from as it moved through the pipeline.- agusmba
What I meant, trying to adapt it to your example, is that if your dataflow C is linking tables from dataflows A and B, but not loading them, when you deploy C on your TEST environment, the linked-but-not-loaded tables will point to A and B on the DEV environment. They are not auto-bound to A and B on the TEST environment unless the "load" tick is set.
