Forum Discussion
Shortcut caching
Which storages does fabric support for shortcut caching? Does it support ADLS Gen2 too? If not then why is this so as one lake is built on top of ADLS Gen2? Can anyone please clarify?
Hello RutujaaKadam,
Shortcut caching is currently supported only for specific shortcut sources. As of today, ADLS Gen2 shortcuts are not supported for shortcut caching.
Although OneLake is built on Azure Data Lake Storage Gen2, shortcut caching is a Fabric-managed optimization rather than a capability inherited from the underlying storage layer. It depends on how Fabric manages metadata, consistency, and cache invalidation for each supported connector.
The lack of ADLS Gen2 support isn't because of a limitation of ADLS itself, but because Fabric hasn't yet enabled shortcut caching for that source type.
If your goal is to improve query performance against ADLS Gen2 data, you may want to consider:
- Materializing frequently accessed data into a Lakehouse Delta table.
- Using shortcuts without caching if near real-time access is more important than performance.
- Keeping an eye on future Fabric updates, as support for additional shortcut sources may be added over time.
If Microsoft has announced plans to extend shortcut caching to ADLS Gen2, I'd be interested to hear from the product team as well.
Best regards,
Omkar Shinde
Microsoft Fabric Enthusiast | Power BI Consultant
💡 If you found this response helpful, please consider giving it a Kudos.
✅ If this resolves your question, please mark it as the Accepted Solution to help others in the community.- There are two different caching concepts that are easy to mix up:1. Shortcut caching2. Intelligent CacheShortcut caching is documented for cross-cloud shortcut sources such as:- Amazon S3- Amazon S3 compatible storage- Google Cloud StorageThe purpose is mainly to reduce repeated reads and egress costs when accessing data across clouds.ADLS Gen2 shortcuts are supported in Fabric, but I do not see shortcut caching documented for ADLS Gen2 in the same way as S3/GCS.For ADLS Gen2, the related optimization is usually Intelligent Cache, but that is different. Intelligent Cache is a Spark-level cache used by Fabric Spark when reading from OneLake or ADLS Gen2 via shortcuts. Microsoft documents that this cache is stored on the local SSD/cache of each Spark node and is used to speed up repeated Spark reads.
Todays capabilities:- S3 / S3-compatible / GCS: shortcut caching is documented- ADLS Gen2: shortcuts are supported, but shortcut caching is not documented in the same way- ADLS Gen2 with Spark: can benefit from Intelligent Cache- Intelligent Cache is Spark-node local cache, not the same as shortcut cachingThe reason is likely architectural and cost-related: shortcut caching is mainly useful for reducing cross-cloud egress and repeated reads, whereas ADLS Gen2 is already Azure-based and Fabric has separate Spark-level caching behavior for OneLake/ADLS reads.🔍Parchitect
Solutions Architect · Microsoft Fabric Specialist💡Helpful? Kudos are appreciated.
✔️Solved? Mark as Solution so others can find it faster. Hi RutujaaKadam ,
Currently, shortcut caching in OneLake supports shortcuts to Google Cloud Storage (GCS), Amazon S3, S3 compatible storage, and shortcuts through the on-premises data gateway (including on-premises S3 shortcuts using Microsoft Entra service principal authentication).
Shortcuts to ADLS Gen2 are not among the supported sources for caching, according to the official Microsoft documentation: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts#caching
Regarding the reason, the documentation does not explain it explicitly, so for me this part is more interpretation than a Microsoft-confirmed fact. The main goal of shortcut caching is to reduce egress costs and latency in cross-cloud scenarios (S3, GCS), in the case of ADLS Gen2, the benefit of caching locally is much smaller, because is Microsoft.
Two additional details from the same documentation that are usually worth keeping in mind:
- Files larger than 1 GB are not cached, even when the source is supported.
- The cache retention period is configurable between 1 and 28 days from the OneLake tab in the workspace settings
If this helped, please consider giving it a Like. If it solved your issue, please mark it as the Accepted Solution to help others facing the same problem.
Thanks
3 Replies
Hello RutujaaKadam,
Shortcut caching is currently supported only for specific shortcut sources. As of today, ADLS Gen2 shortcuts are not supported for shortcut caching.
Although OneLake is built on Azure Data Lake Storage Gen2, shortcut caching is a Fabric-managed optimization rather than a capability inherited from the underlying storage layer. It depends on how Fabric manages metadata, consistency, and cache invalidation for each supported connector.
The lack of ADLS Gen2 support isn't because of a limitation of ADLS itself, but because Fabric hasn't yet enabled shortcut caching for that source type.
If your goal is to improve query performance against ADLS Gen2 data, you may want to consider:
- Materializing frequently accessed data into a Lakehouse Delta table.
- Using shortcuts without caching if near real-time access is more important than performance.
- Keeping an eye on future Fabric updates, as support for additional shortcut sources may be added over time.
If Microsoft has announced plans to extend shortcut caching to ADLS Gen2, I'd be interested to hear from the product team as well.
Best regards,
Omkar Shinde
Microsoft Fabric Enthusiast | Power BI Consultant
💡 If you found this response helpful, please consider giving it a Kudos.
✅ If this resolves your question, please mark it as the Accepted Solution to help others in the community.- There are two different caching concepts that are easy to mix up:1. Shortcut caching2. Intelligent CacheShortcut caching is documented for cross-cloud shortcut sources such as:- Amazon S3- Amazon S3 compatible storage- Google Cloud StorageThe purpose is mainly to reduce repeated reads and egress costs when accessing data across clouds.ADLS Gen2 shortcuts are supported in Fabric, but I do not see shortcut caching documented for ADLS Gen2 in the same way as S3/GCS.For ADLS Gen2, the related optimization is usually Intelligent Cache, but that is different. Intelligent Cache is a Spark-level cache used by Fabric Spark when reading from OneLake or ADLS Gen2 via shortcuts. Microsoft documents that this cache is stored on the local SSD/cache of each Spark node and is used to speed up repeated Spark reads.
Todays capabilities:- S3 / S3-compatible / GCS: shortcut caching is documented- ADLS Gen2: shortcuts are supported, but shortcut caching is not documented in the same way- ADLS Gen2 with Spark: can benefit from Intelligent Cache- Intelligent Cache is Spark-node local cache, not the same as shortcut cachingThe reason is likely architectural and cost-related: shortcut caching is mainly useful for reducing cross-cloud egress and repeated reads, whereas ADLS Gen2 is already Azure-based and Fabric has separate Spark-level caching behavior for OneLake/ADLS reads.🔍Parchitect
Solutions Architect · Microsoft Fabric Specialist💡Helpful? Kudos are appreciated.
✔️Solved? Mark as Solution so others can find it faster. Hi RutujaaKadam ,
Currently, shortcut caching in OneLake supports shortcuts to Google Cloud Storage (GCS), Amazon S3, S3 compatible storage, and shortcuts through the on-premises data gateway (including on-premises S3 shortcuts using Microsoft Entra service principal authentication).
Shortcuts to ADLS Gen2 are not among the supported sources for caching, according to the official Microsoft documentation: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts#caching
Regarding the reason, the documentation does not explain it explicitly, so for me this part is more interpretation than a Microsoft-confirmed fact. The main goal of shortcut caching is to reduce egress costs and latency in cross-cloud scenarios (S3, GCS), in the case of ADLS Gen2, the benefit of caching locally is much smaller, because is Microsoft.
Two additional details from the same documentation that are usually worth keeping in mind:
- Files larger than 1 GB are not cached, even when the source is supported.
- The cache retention period is configurable between 1 and 28 days from the OneLake tab in the workspace settings
If this helped, please consider giving it a Like. If it solved your issue, please mark it as the Accepted Solution to help others facing the same problem.
Thanks