Forum Discussion
What is the practical difference between referencing data via shortcut in ADLS gen2 vs. OneLake?
Hi ,
Thanks for explaining your ask.Yes, there are some features that the Fabric team would be losing by having the data in ADLS Gen2 and reading it via shortcuts vs. having it in OneLake.
Performance: OneLake tables are optimized for performance, especially for queries that involve joins and aggregations. The Fabric team may experience slower performance for queries that involve joins and aggregations if they are reading the data from ADLS Gen2 via shortcuts.
Security: OneLake tables can be secured with role-based access control (RBAC), which makes it easier to manage who has access to the data. The Fabric team would need to manage permissions to the ADLS Gen2 storage account and the Delta Parquet tables if they are reading the data from ADLS Gen2 via shortcuts. This could be more complex than managing permissions to OneLake tables, especially if there are multiple teams accessing the data.
Governance: OneLake tables can be governed with policies, which makes it easier to ensure that the data is used in a compliant manner. The Fabric team would need to develop and implement their own policies to govern the use of the data in ADLS Gen2 if they are reading the data from ADLS Gen2 via shortcuts. This could be more complex than using the built-in policies in OneLake.
In addition to these features, OneLake also offers a number of other advantages, such as:
- Data sharing:OneLake makes it easy to share data with other teams and applications, both within and outside of your organization.
- Data governance:OneLake provides a number of built-in data governance features, such as RBAC, auditing, and data lineage.
- Data quality:OneLake provides a number of data quality features, such as data profiling and data cleansing.
Overall, OneLake is a more comprehensive and robust data management platform than ADLS Gen2. If you are planning to use Fabric to access the data, it is recommended to use OneLake to store and manage the data.
If the Fabric team needs the best performance, security, and governance features, then the data can be stored in OneLake. However, if the Fabric team is only concerned with reading the data, and does not need the additional features of OneLake, then it is possible to read the data from ADLS Gen2 via shortcuts.
Hope this helps. Please let me know if you have any further queries.
I have attached some links. Please refer them for more information:
https://www.infoworld.com/article/3704608/understanding-onelake-and-lakehouses-in-microsoft-fabric.html
https://radacad.com/saying-yes-to-fabric-onelake-shortcuts-no-to-duplicate#:~:text=For%20example%2C%20a%20Shortcut%20can,ADLS%20Gen2%20or%20Amazon%20S3.
I believe you are correct, however I have not seen any documentation that clarifies if Direct Lake Mode in Power BI is possible when leveraging v-ordered Delta Parquet tables via shortcut to ADLS Gen2, or if the v-ordered Delta Parquet tables must truly live in OneLake.
Closest thing I have found is the below graphic seems to indicate that as long as you have a a Fabric/Premium capacity, a Lakehouse, and v-ordered Delta Parquet tables, it should work.
Thanks for your response. The scenario is that we have two teams: one on Fabric and another on Azure. They want to share the same set of data. Does it make any difference to the Fabric team if that data is in ADLSgen2 and read via shortcuts, or if it is in OneLake? In other words is the Fabric team losing any features by having that data in ADLSgen2 vs. having it in OneLake? Let me know if you need further clarification.
Hi ,
Thanks for explaining your ask.
Yes, there are some features that the Fabric team would be losing by having the data in ADLS Gen2 and reading it via shortcuts vs. having it in OneLake.
Performance: OneLake tables are optimized for performance, especially for queries that involve joins and aggregations. The Fabric team may experience slower performance for queries that involve joins and aggregations if they are reading the data from ADLS Gen2 via shortcuts.
Security: OneLake tables can be secured with role-based access control (RBAC), which makes it easier to manage who has access to the data. The Fabric team would need to manage permissions to the ADLS Gen2 storage account and the Delta Parquet tables if they are reading the data from ADLS Gen2 via shortcuts. This could be more complex than managing permissions to OneLake tables, especially if there are multiple teams accessing the data.
Governance: OneLake tables can be governed with policies, which makes it easier to ensure that the data is used in a compliant manner. The Fabric team would need to develop and implement their own policies to govern the use of the data in ADLS Gen2 if they are reading the data from ADLS Gen2 via shortcuts. This could be more complex than using the built-in policies in OneLake.
In addition to these features, OneLake also offers a number of other advantages, such as:
- Data sharing:OneLake makes it easy to share data with other teams and applications, both within and outside of your organization.
- Data governance:OneLake provides a number of built-in data governance features, such as RBAC, auditing, and data lineage.
- Data quality:OneLake provides a number of data quality features, such as data profiling and data cleansing.
Overall, OneLake is a more comprehensive and robust data management platform than ADLS Gen2. If you are planning to use Fabric to access the data, it is recommended to use OneLake to store and manage the data.
If the Fabric team needs the best performance, security, and governance features, then the data can be stored in OneLake. However, if the Fabric team is only concerned with reading the data, and does not need the additional features of OneLake, then it is possible to read the data from ADLS Gen2 via shortcuts.
Hope this helps. Please let me know if you have any further queries.
I have attached some links. Please refer them for more information:
https://www.infoworld.com/article/3704608/understanding-onelake-and-lakehouses-in-microsoft-fabric.html
https://radacad.com/saying-yes-to-fabric-onelake-shortcuts-no-to-duplicate#:~:text=For%20example%2C%20a%20Shortcut%20can,ADLS%20Gen2%20or%20Amazon%20S3.
Thank you! This is exactly the type of response I was looking for.