Forum Discussion
Fabric CI/CD using Python library: Lakehouse deploys but ADLS Gen2 Shortcut fails
Hi All,
I'm using Azure DevOps with the fabric-cicd library to promote Fabric items from a Dev branch to a Test workspace.
Deployment flow:
Lakehouse publishes successfully.
Notebook deployment works.
Shortcut publishing fails.
Error Publishing Lakehouse 'TargetLakehouse' Shortcuts
Deployment failed: Failed to publish 'SampleShortcut' for lakehouse TargetLakehouse
The Lakehouse is created correctly in the target workspace before this step.
The shortcut points to ADLS Gen2 using a workspace connection. I’m using environment-based parameter replacement for:
Lakehouse ID
Workspace ID
ADLS connection ID
Storage account URL
Subpath
append_feature_flag("enable_shortcut_publish") is enabled.
According to the documentation, a Lakehouse must exist in the target workspace before deployment (Correct me if im wrong). I manually created a Lakehouse in a TEST workspace and then deployed from the branch. Deployment works for notebooks and the Lakehouse itself, but shortcuts are not being deployed.
Both source and target workspaces have the correct access permissions.
I also deleted the Lakehouse in TEST and tried deploying again from a DEV workspace (connected to the branch) to the TEST workspace. Same result notebooks and Lakehouse deploy successfully, but shortcuts do not.
Questions:
1.Is ADLS Gen2 shortcut deployment fully supported in fabric-cicd, or is it still experimental?Or do shortcuts need to be recreated or reconfigured manually in the target workspace?
Any guidance appreciated. Thanks!
Thanks everyone for the suggestions, v-dineshya and MJParikh.
The issue is now resolved. The root cause was that the Service Principal used by our Azure DevOps pipeline did not have permission on the ADLS Gen2 workspace connection in the target (TEST) workspace.
After granting the Service Principal access to the ADLS connection, the Lakehouse shortcut deployment worked successfully with enable_shortcut_publish.
4 Replies
Hi CloudVasu,
The issue is not related to your pipeline setup. ADLS Gen2 shortcut deployment via fabric-cicd is partially supported.
A few things to check which you should probably investigate are:
- Connection binding at the target: The ADLS Gen2 connection id in your parameter file should already exist in the TEST workspace and should already be authorized. fabric-cicd does not provision connections; therefore, these should already exist. If the connection id is already correct in your parameter file, then perhaps the connection itself is not yet trusted in the target workspace. In this case, the publish of the shortcut would fail silently.
- Shortcut definition format: Another thing to check is whether the format of the shortcut definition in the repository matches the expected format of the JSON expected by fabric-cicd. The connection id in the JSON in the repository should refer to the workspace-level connection, not personal or gateway connection. This is probably why you are seeing the 'Failed to publish' error without much detail.
- Feature flag scope: Lastly, although you have enabled the append_feature_flag("enable_shortcut_publish"), ensure that this feature flag is enabled before calling deploy_lakehouse_shortcuts(). It is enabled at the pipeline level, which is correct. However, placement does matter.
So, to answer your question, ADLS Gen2 shortcut deployment via fabric-cicd is supported; therefore, the issue is probably the connection.
Recommended next steps:
Manually create ADLS Gen2 connection in the TEST workspace, obtain connection id, add to environment parameter file for TEST workspace, and redeploy application.Thank you!
Proud to be a Super User!
📩 Need more help?
✔️ Don’t forget to Accept as Solution if this guidance worked for you.
💛 Your Like motivates me to keep helping
Thank you for the guidance. I have verified that the ADLS Gen2 connection exists in the target workspace and is authorized, and I updated the parameter file as suggested. However, the shortcut deployment still fails.
From the debug logs, it appears that even after passing the parameter file with the correct TEST values, the shortcuts metadata is still showing the original DEV connection details (connectionId, URL, subpath).
For reference, the parameter file (parameter.yml) contains find_replace mappings for lakehouse IDs, connection IDs, and ADLS Gen2 URLs, and it is passed to the FabricWorkspace object using parameter_file_path. The environment variable target_env is set to test, and the deployment code enables enable_shortcut_publish before calling publish_all_items(). Despite this, the shortcut metadata is still showing DEV values during the deployment logs.
It seems like the parameter replacement is not taking effect for the shortcut definitions, which could be causing the publish failure. Could you advise what else we should check or any known limitations with fabric-cicd for ADLS Gen2 shortcuts that might explain this behavior
Hi CloudVasu ,
Thank you for reaching out to the Microsoft Community Forum.
Please try below things to unblock your pipeline.
1. Pre re-write shortcuts.metadata.json before calling publish_all_items() (stays inside fabric‑cicd). Because find_replace doesn’t apply to Lakehouse content, run a tiny pre‑deploy step in your Azure DevOps pipeline that opens the shortcut JSON files for each Lakehouse and rewrites:
connectionId --> TEST connection’s GUID
ADLS URL / account host --> TEST value
subPath or any environment‑specific fieldsThen run publish_all_items() with enable_shortcut_publish.
Note: While iterating, you can set continue_on_shortcut_failure to avoid blocking the entire deployment if a particular shortcut fails, then remove it once stable.
2. Post‑deploy creation via REST / Notebook / Pipeline, since Lakehouse shortcuts aren’t tracked by Git or Deployment Pipelines, Provision the Lakehouse first, then call the REST API to create/update the shortcuts with target‑specific Connection + ADLS info after the item exists; or trigger a tiny Fabric Data Pipeline that runs a Notebook to create shortcuts from a metadata table.
Please refer below links,
Solved: Re: Shortcuts not supported by deployment pipeline... - Microsoft Fabric Community
Manage a lakehouse with the REST API - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
Thanks everyone for the suggestions, v-dineshya and MJParikh.
The issue is now resolved. The root cause was that the Service Principal used by our Azure DevOps pipeline did not have permission on the ADLS Gen2 workspace connection in the target (TEST) workspace.
After granting the Service Principal access to the ADLS connection, the Lakehouse shortcut deployment worked successfully with enable_shortcut_publish.
