Forum Discussion
Can local PySpark access OneLake using ABFSS paths?
Hello dead_DE
You can use local spark to access your OneLake storage, however you'd have to use a service principal. These are the prerequisites and the steps involved -
1. Make sure your Fabric tenant allows external apps
Your Fabric admin must enable:
- Users can access data stored in OneLake with apps external to Fabric
- (For SPs) Service principals can call Fabric public API
Then grant your service principal Contributor (or above) to the Fabric workspace.
2. Collect the OneLake ABFSS path
OneLake uses ADLS Gen2‑compatible URIs. The account name is always onelake, and the filesystem is your workspace name (or GUID). Typical pattern:
abfss://<workspaceName or workspaceGUID>@onelake.dfs.fabric.microsoft.com/<lakehouseName or itemGUID>.lakehouse/Files/3. Ensure your local Spark has the ABFS connector
You need Hadoop’s hadoop-azure (ABFS) + azure-storage bits on the classpath.
4. Create a Microsoft Entra service principal
Record Tenant (Directory) ID, Client ID (Application ID), and Client Secret, and grant the SP access to your Fabric workspace
5. Configure Spark for ABFS OAuth against OneLake hostpyspark \ --conf "fs.azure.account.auth.type.onelake.dfs.fabric.microsoft.com=OAuth" \ --conf "fs.azure.account.oauth.provider.type.onelake.dfs.fabric.microsoft.com=org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider" \ --conf "fs.azure.account.oauth2.client.id.onelake.dfs.fabric.microsoft.com=<APP_CLIENT_ID>" \ --conf "fs.azure.account.oauth2.client.secret.onelake.dfs.fabric.microsoft.com=<APP_CLIENT_SECRET>" \ --conf "fs.azure.account.oauth2.client.endpoint.onelake.dfs.fabric.microsoft.com=https://login.microsoftonline.com/<TENANT_ID>/oauth2/token"Kindly accept this solution if it solves your problem.
Hello dead_DE
Here's how you can create a service principle or SPN. If there's a team who creates SPN for you in your organisation, they'd know this.
- Go to Entra ID portal
> Applications > App registrations > New registration - Note the:
- Client ID (App ID)
- Tenant ID
- Client Secret
A tenant setting must be changed for SPN to access a Fabric workspace.
Tenant Settings > Developer Settings > “Service principals can use Fabric APIs”
Next you need to assign the SPN Contributor access to your workspace
In Fabric:
- Open the Workspace
- Select Manage access
- Click Add people or groups
- Search for your App Registration name (NOT the GUID!)
- SPNs show up by the registered application name
- Assign an appropriate role:
- Contributor (usually recommended)
Once both the steps are done, you can follow the message I posted on this thread earlier. By assigning the SPN Contributor access to the workspace allows it to write to OneLake.
- Go to Entra ID portal
Following the guide above on creating a SPN i was able to get set up with a App cleint id and and app client secret.
My container running spark needed some configuring as well. I had to set it up with Delta Lake Jars and Azure Blob Storage configurations.
Full Example of how i got this to workfrom pyspark.sql import SparkSession # Create Spark session with Java 17 compatibility spark = (SparkSession.builder .appName("OneLakeAccess") .master("local[*]") .config("spark.jars.packages", "org.apache.hadoop:hadoop-azure:3.3.4,io.delta:delta-spark_2.12:3.1.0") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .config("spark.hadoop.fs.abfss.impl", "org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem") .config("spark.hadoop.fs.abfs.impl", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem") .getOrCreate()) conf = spark.sparkContext._jsc.hadoopConfiguration() conf.set("fs.azure.account.auth.type.onelake.dfs.fabric.microsoft.com", "OAuth") conf.set("fs.azure.account.oauth.provider.type.onelake.dfs.fabric.microsoft.com", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") conf.set("fs.azure.account.oauth2.client.id.onelake.dfs.fabric.microsoft.com", APP_CLIENT_ID) conf.set("fs.azure.account.oauth2.client.secret.onelake.dfs.fabric.microsoft.com", APP_CLIENT_SECRET) conf.set("fs.azure.account.oauth2.client.endpoint.onelake.dfs.fabric.microsoft.com", f"https://login.microsoftonline.com/{TENANT_ID}/oauth2/token") df = spark.read.csv(<abfss_path_to_fabric_resource_in_lakehouse>) df = spark.read.format("delta").load(<abfss_path_to_fabric_delta_table_in_lakehouse>)I am now able to connect to my fabric Lakehouse Files and Delta Tables with ABFSS paths from my spark container.
I have personally used cli to login to Fabric Tenant and used Azure.Identity to authenticate and this has worked for me to connect and test things in local ,though it may not be suitable for production deployments .
az login --allow-no-subscriptions --tenant <TenantId>
from azure.identity import DefaultAzureCredential
delta_token = DefaultAzureCredential().get_token("https://storage.azure.com/.default").token
storage_options = {"bearer_token": delta_token, "use_fabric_endpoint": "true"}
DELTA_TABLE_PATH: str = '<complete abfss path of the lakehouse in Fabric>'
df = DeltaTable(DELTA_TABLE_PATH, storage_options=storage_options)
print(df.to_pandas().head(10))If this code helps ,Accept as a Solution to help others as well
Hi AparnaRamakris I agree, connecting to Fabric OneLake from local Python (Pandas) is not an issue - it works! I think the user was trying to connect local Spark to Fabric, which is slightly different.