Using parameters for workspace and dataflow breaks lineage

[valima]

Hi,

Our Power BI setup is multi environment i.e. we have DEV, UAT and PROD environement. We develop all artifacts (dataflows, datasets, reports) on the dev env and are using deployment pipelines to move those artifacts to UAT and then Prod. I was hoping to use parameters to change the data sources across environment but I am facing a fundamental issue. I lose the lineage view when I use parameters for workspace and dataflow IDs

Let me give an example. 

DEV

Workspace 1 -> Dataflow1 -> DataSourceDev

Workspace 2 -> Dataflow2  -> Dataflow1  (This uses Dataflow1)

What is see when I look at the queries in Dataflow2 is that they have hardcoded workspaceID and dataflowID. Lets say I want to promote this to a UAT env. My understanding is the deployment pipeline will jsut push the dataflow into a new workspace and a new dataflow ids for that env as I need the datasources to be pointed to UAT ones and not DEV.

UAT

Workspace 3 -> Dataflow3 -> DataSourceUAT

Workspace 4 -> Dataflow4  -> ?? 

Deployment pipeline 1

Workspace 1 -> Workspace 3

Deployment pipeline 2

Workspace 2 -> Worspace 4

I am assuming Dataflow4 will still be pointied to Dataflow1 from DEV as the IDs are hardcoded by default by Power BI. So in order to change it to point the Workspace3 -> Dataflow3 I was trying to use parameters for workspace ID and dataflow ID which I can set in the deployment pipeline. But doing that means I lose lineage view. Has anyone come across this situation before and any workaround  that can be done here to show the lineage? Happy to also explore an any alternative to how we can get the dataflows to point to the correct workspaces etc without using parameters

Thanks

[sxe8]

Actually, you don't need to set-up a parameter for the dataflow. If you've read the "Plannine a Power BI Enterprise Deployment" doc; the best recommended practice is to separate your dataflows from your semantic models and report PBI items. That is, you have 6 workspaces in total:

dev, uat & prd for dataflows only

dev, uat & prd for semantic model/reports

Therefore, you have 2 deployment pipeline; 1 for the dataflow workspaces and 1 for reporting workspaces.

So in your dev enviroment you have semantic model that is linked to your dev dataflows (hard coded in).

I have found you don't need a parameter - when you use your deployment pipeline; you should first push your dataflows from dev to uat. Then use the other deployment pipeline to push your semantic model/report from dev to uat. The deployment pipeline recognises that dev = dev and uat = uat so it will pair them.

Then your workspace lineage is kept.

[Anonymous]

Hi @valima ,

I'm sorry I didn't reproduce your problem after my tests. Here's exactly how I did it, I created four workspaces:

Create two parameters for workspace2:

Pipeline 1: Workspace 1 -> Workspace 3

Pipeline 2: Workspace 2 -> Worspace 4

The final lineage view of workspace3 is shown in the picture, is there any inconsistency between my procedure and your description, and can you point it out?

Best Regards,
Zhu

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.

[valima]

Hi Zhu,

Thanks for your reply. Could you please explain how you got WS4 in Test to use the dataflow from WS3 and provide details on what and where the paramters were set?

Thanks

[Anonymous]

Hi @valima ,

I created the dataflow in workspace 4 by doing the following:

Take the workspace id as an example, create a list type parameter, and then filter it.

Best Regards,
Zhu

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.