Forum Discussion
Trouble querying data in ADLS Gen2 from Fabric using a shortcut
Quick update on this: -
I raised a support ticket with Microsoft Fabric Support, and they’ve conducted an in-depth investigation. They were able to replicate the issue when uploading a single Parquet file via the shortcut method above. For this test, I used NYC data from yellow_tripdata_2024-01.parquet.
Interestingly, the issue disappears when multiple files are present in storage or when using other file formats. This confirms that the problem is specific to single Parquet files accessed via shortcuts.
I've now shared this blog with the team, and they have escalated it as a potential bug. Once a solution is found, they will provide an update here.
In the meantime, as a workaround for Parquet files, I'm using: -
df = spark.read.parquet("abfss://container@storage_account.dfs.core.windows.net/*")
display(df)
Hope this helps anyone facing a similar issue!
SukiB
Quick update on this: -
I raised a support ticket with Microsoft Fabric Support, and they’ve conducted an in-depth investigation. They were able to replicate the issue when uploading a single Parquet file via the shortcut method above. For this test, I used NYC data from yellow_tripdata_2024-01.parquet.
Interestingly, the issue disappears when multiple files are present in storage or when using other file formats. This confirms that the problem is specific to single Parquet files accessed via shortcuts.
I've now shared this blog with the team, and they have escalated it as a potential bug. Once a solution is found, they will provide an update here.
In the meantime, as a workaround for Parquet files, I'm using: -
df = spark.read.parquet("abfss://container@storage_account.dfs.core.windows.net/*")
display(df)
Hope this helps anyone facing a similar issue!
SukiB
Hi Sukhminder,
Here’s a workaround that we can offer for this issue as this is a known bug.
In this case, we are placing the file at the container level, which makes it inaccessible due to a known bug in the fabric product. Below is the link to the Bug Item and a supporting document to mention it as a known issue.
Known Issues Tracker - [External ADLS] Shortcuts Load to Tables does not work
Bug Item created on this Bug 1548911 Head call to OneLake for External ADLS is failing at container level.
However, to ease your productivity, we have prioritized this and identified workarounds for testing:
- We need to place the files at the folder level within the container and create a shortcut to that specific folder where our parquet files are located.
and To use the internal storage as a shortcut.
ETA for the bug fix is provided by the end of March.