Read ADLS gen 2 file through Notebook using a Connection

[sureshabey1]

Hi, 

I am trying to run a simple read spark stream query with in fabric notebook. 

I have used workspace identity mechanism for connection creation to storage account.

https://learn.microsoft.com/en-us/fabric/security/workspace-identity-authenticate

from pyspark.sql import SparkSession

from pyspark.sql.functions import col, lit, struct, to_json

from delta.tables import *

spark = SparkSession.builder.getOrCreate()

df = (

    spark.readStream

    .format("json")

    .schema("site_Group string, site string)

    .option("recursiveFileLookup", "true")

    .option("ignoreMissingFiles", "true")

    .option("withEventTimeOrder", "true")

    .load("abfss://landing-zone@xyz_storageaccount.dfs.core.windows.net/Y/")

)

I have given the workspace the azure blob data reader access at the storage account level. 

But still getting access related issue to read the file.

Operation failed: "This request is not authorized to perform this operation.", 403, HEAD

Any help would be appreciated.

[v-echaithra]

Hi @sureshabey1 ,

May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.

Thank you.

[v-echaithra]

Hi @sureshabey1 ,

Thank you @Shreya_Barhate , @deborshi_nag  for your inputs.

We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

Thank you for your patience and look forward to hearing from you.
Best Regards,
Chaithra E.

[Shreya_Barhate]

hi @sureshabey1 ,

Workspace identity works only for scenarios where Fabric uses -workspace identity to authenticate—specifically when connecting OneLake shortcuts, Data Pipelines, Semantic Models, and Dataflows Gen2 (CI/CD) to external data sources.
It does not currently authenticate Spark notebooks for direct file access to ADLS Gen2. For notebooks, you must use either user identity, service principal, or managed identity configured through a linked service with appropriate ACL/RBAC permissions.


Thanks and regards,
Shreya Barhate

[deborshi_nag · Accepted solution]

Hello @sureshabey1 

Since you're using the fully qualified abfs location, I'm guessing you have not created the OneLake shortcut. Follow the Microsoft doc link you shared to create a OneLake shortcut pointing to the folder in your external storage. Use Workspace Identity when creating the shortcut. Once done, attach the notebook to the Lakehouse and use /Files/<shortcut name> instead of the abfs location. 

Secondly, if your storage is behing a firewall, please check you've used Trusted Workspace Access.