Forum Discussion
Solved
Invoke Pipeline (Preview) - Managing Pipeline IDs Across Workspaces
Scenario: We have a master pipeline in Microsoft Fabric that dynamically calls worker pipelines by passing the Pipeline ID (GUID), which is retrieved from a SQL Server database. This setup works per...
Thanks for the input, have applied a workaround to capture the workspace details using REST API and capture that info. in the backend table and utilize that to share it with Invoke Pipeline (Preview) dynamically.
I can think of 2-3 ways of maintaining pipeline ids across workspaces - this is a very similar problem to when you have DEV, TEST, and PROD environments and need to make sure lakehouse ids move properly during deployments.
- Probably the worst case from a maintenance point of view, but have a table in each workspace that holds lookups between pipeline name and pipeline id. Create a sub-pipeline that takes a pipeline name as a parameter and returns a pipeline id. This table could be periodically populated using API calls similar to method 2 without the filtering and output to a table.
- Have a notebook that uses the library sempy_labs and specifically the function sempy_labs.admin.list_items() to get a list of items, filter on Type = DataPipelines. Have the notebook take the pipeline name in as a parameter and output the pipeline id as its exitValue
- Same as 2., just write the API calls yourself. (Mostly for when you can't install sempy_labs). I won't demo this here, but sempy_labs just wraps these API calls so it works the same way.