Forum Discussion

sureshabey1's avatar
sureshabey1
New Member
6 months ago
Solved

Read ADLS gen 2 file through Notebook using a Connection

Hi, 

I am trying to run a simple read spark stream query with in fabric notebook. 

I have used workspace identity mechanism for connection creation to storage account.

https://learn.microsoft.com/en-us/fabric/security/workspace-identity-authenticate

 

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, lit, struct, to_json
from delta.tables import *

spark = SparkSession.builder.getOrCreate()

df = (
    spark.readStream
    .format("json")
    .schema("site_Group string, site string)
    .option("recursiveFileLookup", "true")
    .option("ignoreMissingFiles", "true")
    .option("withEventTimeOrder", "true")
    .load("abfss://landing-zone@xyz_storageaccount.dfs.core.windows.net/Y/")
)

 

I have given the workspace the azure blob data reader access at the storage account level. 

But still getting access related issue to read the file.

 

Operation failed: "This request is not authorized to perform this operation.", 403, HEAD

Any help would be appreciated.

 

Notebook
  • deborshi_nag's avatar
    deborshi_nag
    6 months ago

    Hello sureshabey1 

     

    Since you're using the fully qualified abfs location, I'm guessing you have not created the OneLake shortcut. Follow the Microsoft doc link you shared to create a OneLake shortcut pointing to the folder in your external storage. Use Workspace Identity when creating the shortcut. Once done, attach the notebook to the Lakehouse and use /Files/<shortcut name> instead of the abfs location. 

    Secondly, if your storage is behing a firewall, please check you've used Trusted Workspace Access.

     

4 Replies

  • deborshi_nag's avatar
    deborshi_nag
    Super User
    6 months ago

    Hello sureshabey1 

     

    Since you're using the fully qualified abfs location, I'm guessing you have not created the OneLake shortcut. Follow the Microsoft doc link you shared to create a OneLake shortcut pointing to the folder in your external storage. Use Workspace Identity when creating the shortcut. Once done, attach the notebook to the Lakehouse and use /Files/<shortcut name> instead of the abfs location. 

    Secondly, if your storage is behing a firewall, please check you've used Trusted Workspace Access.

     

  • Shreya_Barhate's avatar
    Shreya_Barhate
    Resolver II
    6 months ago

    hi sureshabey1 ,

    Workspace identity works only for scenarios where Fabric uses -workspace identity to authenticate—specifically when connecting OneLake shortcuts, Data Pipelines, Semantic Models, and Dataflows Gen2 (CI/CD) to external data sources.
    It does not currently authenticate Spark notebooks for direct file access to ADLS Gen2. For notebooks, you must use either user identity, service principal, or managed identity configured through a linked service with appropriate ACL/RBAC permissions.


    Thanks and regards,
    Shreya Barhate

  • v-echaithra's avatar
    v-echaithra
    Community Support
    6 months ago

    Hi sureshabey1 ,

    Thank you Shreya_Barhate , deborshi_nag  for your inputs.

    We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Chaithra E.

  • v-echaithra's avatar
    v-echaithra
    Community Support
    5 months ago

    Hi sureshabey1 ,

    May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.

    Thank you.