Assistance Needed: Issues with PySpark Migration and ADLS Connectivity

Hi BharatiPilla ,
Thanks for using Microsft Fabric Community,

ADLS Gen1 was officially retired on February 29, 2024, and Microsoft recommends migrating to ADLS Gen2, which is fully supported in Microsoft Fabric with better security and performance.

Possible Workaround for ADLS Gen1 (If Still Accessible):

Since mssparkutils.credentials.getToken() isn’t working consistently in Fabric Notebooks (this has been reported by multiple users), you can try manually retrieving an OAuth token using the Azure Identity SDK and accessing ADLS Gen1 via the WebHDFS API:

from azure.identity import DefaultAzureCredential
import requests

# Retrieve OAuth token for ADLS Gen1
credential = DefaultAzureCredential()
token = credential.get_token("https://datalake.azure.net/.default").token

# WebHDFS API call
headers = {"Authorization": f"Bearer {token}"}
adls_url = "https://<your_adls_gen1_account>.azuredatalakestore.net/webhdfs/v1/<your_file_path>?op=OPEN"

response = requests.get(adls_url, headers=headers)

if response.status_code == 200:
print("Data Retrieved Successfully")
else:
print(f"Error: {response.status_code}, {response.text}")

This method was valid before ADLS Gen1’s retirement, but since the service is no longer available, it may not work and could return a 403 Forbidden error.

Here is a similar thread which might help: Get Linked service token in Microsoft Fabric noteb... - Microsoft Fabric Community

Since ADLS Gen1 is fully retired, the best approach is to migrate your data to ADLS Gen2, which works seamlessly with Fabric.

Here’s Microsoft’s guide on migration: Migrate ADLS Gen1 to ADLS Gen2.

If you require further assistance, we recommend raising a support ticket with Microsoft for more in-depth help.

Hope this helps. Pleas reach out for further assistance.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated

Regards,
Vinay.