Deployment with fabric-cicd: How to handle cross-workspace references in pipelines?

[ObungiNiels]

Hi fabricators, 

I am using the fabric-cicd package in all my projects and generally am a big fan of the library. 

In a new project, I am facing an issue due to this client's workspace set up. The medallion architecture is split into multiple workspaces, one more bronze and silver and another one for gold. 

My main orchestration pipeline calls three sub-orchestration pipelines which execure the data movement steps (source-bronze, bronze-silver and silver-gold). This main orchestration pipeline resides in a separate workspace from the silver-gold sub-orchestration pipeline.

My problem: When deploying with the fabric-cicd package, the binding of the item id to the workspace in an invoke pipeline activity only works, when the invoked pipeline artifact resides in the same workspace as the pipeline which has the invoke pl activity. 

For the described scenario, the invoke pipeline activity looks like this after deploying: 

The workspace reference is changed correctly (workspace id mapping is included in the parameter.yml file), however the field pipeline shows the ID of the pipeline from my dev environment. As mentioned, this is only a problem for a cross-workspace invoke scenario. 

My question: Do you know a way of solving my problem without having to add item ID of pipelines to the parameter.yml file? I do not want to add item IDs since these change easily with time and require a two-step deployment in the initial phase. 

Appreciate your input, as always! 🙂 

[KevinChant]

Have you already looked into variable libraries to resolve this?

Fabric Application Lifecycle Management Variable Library - Microsoft Fabric | Microsoft Learn 

[ObungiNiels]

Hi @KevinChant ,

thanks for your answer! I am using the variable library, but using that would require me to store the item ID in the variable lib, which poses the same issue as storing it in the parameter.yml file. When the item id changes (because the item is recreated), I'll have to change the reference. 

It looks like I will have to go down this path, since no other solution presents itself, but I was hoping to find a more elegant way of handling this. 🙂 

[v-saisrao-msft]

Hi @ObungiNiels,

This is a current limitation in Microsoft Fabric by design, and cross-workspace Invoke Pipeline references are not automatically resolved at this time.
Could you please confirm if the workaround you mentioned has resolved the issue on your end?

Thank you.

[ObungiNiels]

Hi @v-saisrao-msft , yes thank you for confirming. I have resolved my implemention by adding the single invoke pipeline activity to my find_replace script, which is a good, temporary workaroung for me. 

[Vinodh247 · Accepted solution]

i think you have hit a current limitation of fabric-cicd. invoke pipeline bindings are only auto patched when the referenced pipeline artifact exists in the same workspace. Cross workspace references are not resolved, so the deployment does exactly what you saw, it updates the workspace but leaves the pipeline ID behind from dev. Afaik, until fabric cicd supports name-to-ID resolution across workspaces, the safest and simplest route is a single-control-workspace for orchestration.

[ObungiNiels]

Hi @Vinodh247 ,

thanks for your reply. Unfortunately, changing the workspace architecture is not an option. Since it is only one item ID, I am considering to add this to my var lib / parameter.yml file until I find a better solution. 

Hoping for cross-workspace binding to be added to the fabric-cicd package in the future. 🙂