Assistance Needed: Issues with PySpark Migration and ADLS Connectivity

[BharatiPilla]

Hi Team,

I am working on rewriting a Scope script (USQL) into PySpark and have encountered a few challenges:

1. While there is a shortcut available for ADLS Gen2, I couldn't find similar support for ADLS Gen1. Is there an alternative?

2. When trying to connect from Notebooks, I noticed that, unlike Synapse, there is no linked service available for authentication. Additionally, I am facing issues retrieving a token, as the basic mssparkutils functionality isn't working.

Could someone assist me in resolving these issues?

Thanks,
Bharati

[v-veshwara-msft · Accepted solution]

Hi @BharatiPilla ,
Thanks for using Microsft Fabric Community,

ADLS Gen1 was officially retired on February 29, 2024, and Microsoft recommends migrating to ADLS Gen2, which is fully supported in Microsoft Fabric with better security and performance.

Possible Workaround for ADLS Gen1 (If Still Accessible):

Since mssparkutils.credentials.getToken() isn’t working consistently in Fabric Notebooks (this has been reported by multiple users), you can try manually retrieving an OAuth token using the Azure Identity SDK and accessing ADLS Gen1 via the WebHDFS API:

from azure.identity import DefaultAzureCredential
import requests

# Retrieve OAuth token for ADLS Gen1
credential = DefaultAzureCredential()
token = credential.get_token("https://datalake.azure.net/.default").token

# WebHDFS API call
headers = {"Authorization": f"Bearer {token}"}
adls_url = "https://<your_adls_gen1_account>.azuredatalakestore.net/webhdfs/v1/<your_file_path>?op=OPEN"

response = requests.get(adls_url, headers=headers)

if response.status_code == 200:
print("Data Retrieved Successfully")
else:
print(f"Error: {response.status_code}, {response.text}")

This method was valid before ADLS Gen1’s retirement, but since the service is no longer available, it may not work and could return a 403 Forbidden error.

Here is a similar thread which might help: Get Linked service token in Microsoft Fabric noteb... - Microsoft Fabric Community

Since ADLS Gen1 is fully retired, the best approach is to migrate your data to ADLS Gen2, which works seamlessly with Fabric.

Here’s Microsoft’s guide on migration: Migrate ADLS Gen1 to ADLS Gen2.

If you require further assistance, we recommend raising a support ticket with Microsoft for more in-depth help.

Hope this helps. Pleas reach out for further assistance.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated

Regards,
Vinay.

[v-veshwara-msft]

Hi @BharatiPilla ,

Just checking in to see if your query has been resolved and whether any of the responses were helpful. If so, kindly consider marking the relevant reply as the Accepted Solution to assist others who may have a similar question.

Additionally, could you please confirm whether the issue has been addressed through the support ticket with Microsoft?

If the issue is now resolved, we’d greatly appreciate it if you could share any key insights or the resolution here for the benefit of the wider community.

Should you need further assistance in the future, please feel free to create a new post in the Microsoft Fabric Community Forum. We’ll be glad to support you.

We may consider closing this thread for now.

Thank you for your time and participation.