Intermittent issues with accessing files over ADLSv2 shortcut using "File API path" from a notebook

[1569hypernova]

Ive been using a notebook to process files from a ADLSv2 shortcut. The data on ADLSv2 is created by Azure Synapse Link for Dataverse with Azure Data Lake. The issue is that i cant access files over "File API path". Ive reproduced the issue below using %%sh but its the same using pyspark. The issue seems to resolve itself after a while and\or maybe after ive interacted with the shortcut, I'm not entierly sure. See the second screenshot from a couple of minutes later when it works again all of a sudden.

"Dataverse...5435" is mounting point for the shortcut.

Reading files into a dataframe using the "relative path for spark" always work of the bat. Ie

Have anyone experiences similar issues? Is there some known limitation or current bug? Any ideas on a workarround?

[V-yubandi-msft]

Hi @1569hypernova ,

We noticed we haven't received a response from you yet, so we wanted to follow up and ensure the solution we provided addressed your issue. If you require any further assistance or have additional questions, please let us know.

Your feedback is valuable to us, and we look forward to hearing from you soon.

Thank You.

[V-yubandi-msft]

Hi @1569hypernova ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution we provided for your issue worked for you  or let us know if you need any further assistance?

Your feedback is important to us, Looking forward to your response. 

Thank You.

[V-yubandi-msft]

Hi @1569hypernova ,

We wanted to touch base as we haven't received a response from you yet. We hope the solution provided was useful. If you require further assistance or have any more questions, don't hesitate to reach out. Your feedback is invaluable, and we look forward to your response.

Thank You.

[V-yubandi-msft]

Hi @1569hypernova ,

Thank you for sharing your workaround. While your approach works well, we suggest considering a more efficient retry mechanism using Exponential Backoff.

• Instead of retrying at fixed intervals you can gradually increase the wait time between retries ( 5s → 10s → 20s → 40s).
• This method reduces unnecessary delays and enhances reliability when accessing the shortcut.
• It also incorporates a maximum retry limit to prevent infinite loops.
• If the shortcut remains inaccessible, please check permissions, cluster initialization, and the sync timing of the Synapse Link.

If my answer addressed your query, please mark it as the Accepted Solution so that others can easily find the solution. I would also appreciate a 'Kudos' if you found my response helpful.

[1569hypernova]

Thanks @nilendraFabric for the idea and links. Unfortionally mssparkutils experiences the same issue. Non of the links seems to be the same issue, at least on a surface level.

However, mssparkutils.fs.ls returns an empty list so for now I just made a loop to wait for my shortcut to be accessible if\when the issue occurs as a workarround. 

df_mnt = mssparkutils.fs.ls(f"file://{mssparkutils.fs.getMountPath('/default/Files/synapse_link/')}")

while not df_mnt:

    print("Waiting 10 seconds for shortcut to mount...")

    sleep(10)

    df_mnt = mssparkutils.fs.ls(f"file://{mssparkutils.fs.getMountPath('/default/Files/synapse_link/')}")

[nilendraFabric · Accepted solution]

Hello @1569hypernova 

path = mssparkutils.fs.getMountPath("/your_shortcut_path")
files = mssparkutils.fs.ls(path)

Please check if this working for you.

Not officially known issue but other user have reported similar type of issue 

https://www.reddit.com/r/MicrosoftFabric/comments/1ff5vd9/anyone_else_having_intermittent_errors_wit...

https://community.fabric.microsoft.com/t5/Fabric-platform/Fabric-Lakehouse-shortcut-to-ADLS-gen2-not...

Please accept the answer if this is helpful and give kudos


Thanks