Deleting lakehouse shortcut does not fully remove shortcut

[belshekheiby]

I have a few Link to Fabric Lakehouses connected to different Dataverse environments that I was creating shortcuts from into another lakehouse LH_DP in another workspace. I programmatically deleted the shortcuts in LH_DP and tried to recreate them as I was changing environments but when I try to recreate the shortcut I keep getting a 400 Bad Request error message:

Error: {"requestId":"xyz","errorCode":"BadRequest","moreDetails":[{"errorCode":"ShortcutNotFound","message":"Shortcut table_name is not found"}],"message":"The request could not be processed due to missing or invalid information"}

I have been running this approach in notebooks for months but now I'm not able to recreate the shortcuts anymore.

One thing I noticed was that if I run the notebookutils.fs.ls command I can still see the parent directory of the shortcut but it is empty. 

Any assistance would be appreciated.

[belshekheiby · Accepted solution]

So I figured out what the problem was. The OneLake Delete Shortcut API endpoint has quietly changed behaviour to delete the shortcut but still keeping the storage folder in the file system of the lakehouse under the hood.

https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/delete-shortcut?tabs=HTTP

This makes this API endpoint redundant if anyone is planning on deleting and recreating shortcuts with the exact same names. The workaround is to delete the files and folders using notebookutils.fs.rm in a notebook for example which deletes the shortcut and the corresponding files.

[PaulKn]

I have experienced a similar problem today - I had a function to delete a shortcut using the API and when I ran the command it never gave a response (normally I get a success response in 30-40 seconds) and just carried on running. I could see the shortcut had disappeared but I got an error when trying to recreate it through the API. When I tried to recreate it through the front end it gave it a "_1" suffix. I tried to rename it using the new manage shortcut functionality in the front-end to the desired name and that gave me "There is a currently a lease on the resource and no lease ID was specified in the request" error, which brought me to this thread.

Can I confirm what you did to resolve the situation with a shortcut you had deleted through the API that you wanted to rebuild? I have tried to use notebookutils.fs.rm by doing the following:

1) Running notebookutils.fs.ls("Tables/") which confirmed there was an underlying folder still there for my shortcut

2) Ran notebookutils.fs.rm('Tables/[shortcutname]', True) (where [shortcutname] is my shortcutname) to try and delete it but I get the error "There is currently a lease on the resource and no lease ID was specified in the request" again. 

This seems like a serious bug if you are unable to recreate a shortcut with the same name after deleting it.

Thanks

[belshekheiby]

Hi,

If you had already deleted the shortcut throught the Delete Shortcut API endpoint then I'm afraid there's no workaround. The file/folder representing the shortcut will forever be in the lakehouse and will have the "lease" on it and you cannot delete it.

I had to backup my lakehouse, delete it altogether then recreate it and reload the tables (physical tables, not shortcuts) from the backup by using a data pipeline. I believe all of those recent changes were introduced because shortcuts can now be source controlled.

My advice would be to disregard deleting shortcuts and only rely on the ShortcutConflictPolicy=CreateOrOverwrite URI parameter in the create shortcut API endpoint. This helps overwrite shortcuts that are checked into source control e.g. dev environment shortcuts are checked into git but you need to overwrite them in your UAT environment with the UAT shortcuts.

If you REALLY want to delete shortcuts then run notebookutils.fs.rm('Tables/[shortcutname]', True)  in your notebook instead of the delete shortcut API - it is practically useless now.

[PaulKn]

Hi, I raised a support ticket and this is a bug. A fix is going to be rolled out.

There is a workaround in the meantime for shortcuts that have not been fully deleted through the API - you can connect to the lakehouse in Azure Storage Explorer (Integrate OneLake with Azure Storage Explorer - Microsoft Fabric | Microsoft Learn), find the file for the shortcut that has not been fully deleted and then Right click on it and choose "Break Lease". You can then delete the lingering file and recreate the shortcut.

[v-vpabbu]

Hi @belshekheiby,

Glad your issue has been resolved!
I suggest you to accept your own post as the solution — it will help other community members facing similar problems to find the answer faster.

Regards,

Vinay Pabbu

[rosha_rosha]

Hi,

Steps to Resolve the Issue

1. Verify Dataverse Environment Configuration:
• Ensure the new Dataverse environment is properly linked to the source Lakehouse using the "Link to Microsoft Fabric" feature in the PowerApps maker portal. This step syncs the Dataverse tables to the Fabric Lakehouse, making them available for shortcuts [Ref web ID: 0].
• Check that the Dataverse environment and Fabric Capacity are in the same Azure geographic region. You can verify this in the Fabric workspace settings and the Dataverse environment settings [Ref web ID: 2].
• Confirm that the account or service principal used to create shortcuts has the System Administrator role in the new Dataverse environment and admin access to the Fabric workspace [Ref web ID: 0].
2. Clean Up Residual Metadata:
• Since notebookutils.fs.ls shows an empty parent directory, there might be leftover metadata causing conflicts. Use the Fabric UI to manually inspect the LH_DP Lakehouse and delete any empty directories or broken shortcuts under the Tables or Files folder where the shortcuts were created.
• If the UI doesn’t allow deletion, try using the notebookutils.fs.rm command to programmatically remove the empty directory:

from notebookutils import mssparkutils

mssparkutils.fs.rm("abfss://<workspace>@<lakehouse>.dfs.fabric.microsoft.com/<lakehouse_name>/Tables/<shortcut_directory>", True)

Replace the placeholders with your workspace, Lakehouse, and directory names.

1. Recreate the Source Lakehouse Link:
• If the Dataverse environment change caused the issue, you might need to re-link the source Lakehouse to the new Dataverse environment. Go to the PowerApps maker portal, navigate to the new Dataverse environment, and use the "Link to Microsoft Fabric" feature to create a new Lakehouse or update the existing one. This will regenerate the shortcuts for the Dataverse tables in the source Lakehouse [Ref web ID: 1].
2. Manually Create a Shortcut to Test:
• Before retrying programmatically, use the Fabric UI to manually create a shortcut in LH_DP pointing to a table in the source Lakehouse. This will help confirm whether the issue is with the API or the environment setup.
• In the LH_DP Lakehouse, navigate to the Tables folder, right-click, and select "New Shortcut." Choose the source Lakehouse, select a table, and create the shortcut. If this fails with the same error, the issue is likely with the source Lakehouse or Dataverse environment.
3. Check for Organization Movement:
• If your Dataverse environment was moved to a new storage location, this could be causing the issue. The web information notes this as a known problem with no immediate workaround other than avoiding organization movement or contacting Microsoft Support [Ref web ID: 20]. If this applies, you may need to create a new Dataverse environment in the original region and relink it to Fabric.
4. Inspect Notebook Code for Shortcut Creation:
• Review the code you’re using to recreate the shortcuts. Fabric supports creating shortcuts via the REST API, but the web information indicates that programmatic creation can be less reliable than the UI [Ref web ID: 11]. Ensure your API call includes the correct target path, authentication, and table name. A sample API call to create a shortcut might look like this:

curl -X POST "https://api.fabric.microsoft.com/v1/workspaces/<workspace_id>/lakehouses/<lakehouse_id>/shortcuts" \

-H "Authorization: Bearer <token>" \

-H "Content-Type: application/json" \

-d '{

  "name": "table_name",

  "path": "Tables",

  "target": {

    "oneLake": {

      "workspaceId": "<source_workspace_id>",

      "lakehouseId": "<source_lakehouse_id>",

      "path": "Tables/table_name"

    }

  }

}'

If the API call fails with the same 400 error, double-check the target path and ensure the table exists in the source Lakehouse.

1. Contact Microsoft Support:
• If the above steps don’t resolve the issue, it’s possible there’s a deeper problem with the Fabric service, such as a bug introduced in a recent update. The web information suggests collecting details like the workspace ID, region, and support request number for troubleshooting [Ref web ID: 13]. Open a support ticket with Microsoft, providing:
• The requestId from the error message (xyz in your case).
• The workspace ID and region for LH_DP and the source Lakehouses.
• Details of the Dataverse environment change.

[belshekheiby]

Hi,

Thank you for your reply.

Here are steps that I have already tried to debug this problem, some of them align with the steps above:

1. Manually created a shortcut
   
   • I have tried to manually create a shortcut from the Fabric UI in the lakehouse and was able to successfully create multiple ones. I think this eliminates the possibility of a broken link either in Dataverse or the linked Lakehouses.
2. Use mssparkutils.fs.rm to manually remove the residual files/folder
   • When I tried this I received a Operation failed: "Precondition Failed", 412.....LeaseIdMissing, "There is currently a lease on the resource and no lease ID was specified in the request
3. Tried running the shortcut creation code via REST API multiple times, with different parameter values for ShortcutConflictPolicy. Received the same error message again.

I had this code running for months. It was deleting and recreating shortcuts on the fly since it was developed before introducing the ShortcutConflictPolicy=CreateOrOverwrite. The main issue here is the behaviour of shortcut deletion which leaves these residue files in the lakehouse which has never been the case before the last Fabric update this past weekend.