Forum Discussion
Trouble querying data in ADLS Gen2 from Fabric using a shortcut
Quick update on this: -
I raised a support ticket with Microsoft Fabric Support, and they’ve conducted an in-depth investigation. They were able to replicate the issue when uploading a single Parquet file via the shortcut method above. For this test, I used NYC data from yellow_tripdata_2024-01.parquet.
Interestingly, the issue disappears when multiple files are present in storage or when using other file formats. This confirms that the problem is specific to single Parquet files accessed via shortcuts.
I've now shared this blog with the team, and they have escalated it as a potential bug. Once a solution is found, they will provide an update here.
In the meantime, as a workaround for Parquet files, I'm using: -
df = spark.read.parquet("abfss://container@storage_account.dfs.core.windows.net/*")
display(df)
Hope this helps anyone facing a similar issue!
SukiB
Thanks spaceman127. I've created a brand new storage from sratch, and given permission to the user via IAM. My Fabric username is part of the following: -
Owner, Contributor, Storage Blob Data Contributor and Storage Blob Data Reader.
Ref. the ACL - I've set the Security Principal as follows...
Owner: $superuser - Read, Write, Execute
Owning group: $superuser - Read, Write, Execute
Other - Read, Write, Execute
Still get the same error.
Super UserHello SukiB
Did you Enable hierarchical namespace?
Verify execute permissions on parent folders and read permissions on files in ADLS Gen2’s hierarchical namespace
Thanks Nilendra.
Hierarchical Namespace is enabled. For manage ACL, I've provision everything to Read, Write and Execute - just in case. Still no luch I'm afraid.