Forum Discussion
Trouble querying data in ADLS Gen2 from Fabric using a shortcut
Quick update on this: -
I raised a support ticket with Microsoft Fabric Support, and they’ve conducted an in-depth investigation. They were able to replicate the issue when uploading a single Parquet file via the shortcut method above. For this test, I used NYC data from yellow_tripdata_2024-01.parquet.
Interestingly, the issue disappears when multiple files are present in storage or when using other file formats. This confirms that the problem is specific to single Parquet files accessed via shortcuts.
I've now shared this blog with the team, and they have escalated it as a potential bug. Once a solution is found, they will provide an update here.
In the meantime, as a workaround for Parquet files, I'm using: -
df = spark.read.parquet("abfss://container@storage_account.dfs.core.windows.net/*")
display(df)
Hope this helps anyone facing a similar issue!
SukiB
Solution SageI found a similar post here with some errors while reading multiple CSV files: https://community.fabric.microsoft.com/t5/Fabric-platform/Read-multiple-files-in-Fabric-Notebook/td-p/3744395. Can you try to only include the folder which contains the files, like this:
df = spark.read.format("parquet").load("Files/landing-zone")
Solution SageFurthermore, do you know how the Shortcut Authorization is set up? Did you create the shortcut to ADLS Gen2 yourself, or did you use an existing shortcut? The shortcut is probably set up with a SAS-token, given the error message. Make sure that the SAS-token has sufficient permissions or set up a new shortcut with your user account as authorization and try again.
- Created the Shortcut myself, to Azure Data Lake Storage Gen2 and using Account Key for authorisation.
What's weird is that whether I use Account Key or SAS Authorisation - it gives me the same error...
Spark_System_ABFS_OperationFailed
An operation with ADLS Gen2 has failed. This is typically due to a permissions issue. 1. Please ensure that for all ADLS Gen2 resources referenced in the Spark job, that the user running the code has RBAC roles "Storage Blob Data Contributor" on storage accounts the job is expected to read and write from. 2. Check the logs for this Spark application. Inspect the logs for the ADLS Gen2 storage account name that is experiencing this issue.But, loading from OneDrive in Fabric works - so I know the code is good. In summary...
df = spark.read.parquet("Files/landing-zone-sas/*") --doesn't work
df = spark.read.parquet("Files/landing-zone-accountkey/*") --doesn't work
df = spark.read.parquet("Files/landing-zone-onedrive/*") --works - this is the files being uploaded into onedrive
Any further thoughts - much appreciated.