Forum Discussion
Post PatronHow to integrate Databricks into Fabric
Hi fabricpribeiro,
Yes, Azure Databricks stores delta tables in cloud storage for example ADLS Gen2, while databricks provides the compute and governance layer over that data.
- If you create a shortcut to a delta table in supported storage, it will appear under the Tables section of your fabric lakehouse. Since the data is stored in delta format, the lakehouse automatically recognizes it as a table, synchronizes the schema and we can easily query it.
- If you create the shortcut under Files, it remains as raw files and will not automatically be recognized as a table, they will appear under the Files section.
For permissions, there are two aspects:
- The shortcut connection must have permission to access the underlying storage where the data resides, this is configured through the shortcut’s connection
- Once the shortcut is created, users only need the appropriate permissions in fabric to access it. They don't need the databricks permissions because the shortcut reads directly from ADLS Gen2 through the configured connection.
- If you are using Mirroring for azure databricks(Unity Catalog) instead of an ADLS shortcut, that is different. Mirroring interacts with unity catalog and follows its permissions and supported scenarios.
Regarding data freshness, shortcuts do not copy the data, they read it in place. As changes are written to the underlying delta tables, fabric reads the latest data from the source, so there isn't a separate synchronization schedule like there is with mirroring.
Thanks and regards,
Anjan Kumar Chippa
Hi fabricpribeiro,
- Use Shortcuts when you want to access the data where it already exists, avoid data duplication and always work with the latest data in the source storage.
- Use Mirroring when you want fabric to maintain a managed copy of the supported source data in OneLake so that it can be used across fabric workloads without building custom ingestion pipelines.
For your reference, follow this below documentation:
Microsoft Fabric Mirrored Catalog From Azure Databricks - Microsoft Fabric | Microsoft Learn
Unify Data Sources With OneLake Shortcuts - Microsoft Fabric | Microsoft Learn
Thanks and regards,
Anjan Kumar Chippa
Post PatronAre you saying that databricks writtes by default to azure storage and that we can use fabric to point to those elta tables created by default by databricks into the fabric? where will those land? the file section of my fabric lakehouse or in the tables section? and hwo do permissions work for this shorcuts? do I need permissions on databricks ? or just on fabric so that users can consume the information? maybe there are permissions for fabric to access the databricks files and permissions then in fabric for users to consume the information ? or do we need as well to provide the permissions of the fabric users to the databricks platform? and what is time that data is delivered? near real time?
Community SupportHi fabricpribeiro,
Yes, Azure Databricks stores delta tables in cloud storage for example ADLS Gen2, while databricks provides the compute and governance layer over that data.
- If you create a shortcut to a delta table in supported storage, it will appear under the Tables section of your fabric lakehouse. Since the data is stored in delta format, the lakehouse automatically recognizes it as a table, synchronizes the schema and we can easily query it.
- If you create the shortcut under Files, it remains as raw files and will not automatically be recognized as a table, they will appear under the Files section.
For permissions, there are two aspects:
- The shortcut connection must have permission to access the underlying storage where the data resides, this is configured through the shortcut’s connection
- Once the shortcut is created, users only need the appropriate permissions in fabric to access it. They don't need the databricks permissions because the shortcut reads directly from ADLS Gen2 through the configured connection.
- If you are using Mirroring for azure databricks(Unity Catalog) instead of an ADLS shortcut, that is different. Mirroring interacts with unity catalog and follows its permissions and supported scenarios.
Regarding data freshness, shortcuts do not copy the data, they read it in place. As changes are written to the underlying delta tables, fabric reads the latest data from the source, so there isn't a separate synchronization schedule like there is with mirroring.
Thanks and regards,
Anjan Kumar Chippa
- fabricpribeiro
This is very good reply. One last question, please
When to use one (shortcuts) or the other (mirroring)?
- v-achippa
Hi fabricpribeiro,
- Use Shortcuts when you want to access the data where it already exists, avoid data duplication and always work with the latest data in the source storage.
- Use Mirroring when you want fabric to maintain a managed copy of the supported source data in OneLake so that it can be used across fabric workloads without building custom ingestion pipelines.
For your reference, follow this below documentation:
Microsoft Fabric Mirrored Catalog From Azure Databricks - Microsoft Fabric | Microsoft Learn
Unify Data Sources With OneLake Shortcuts - Microsoft Fabric | Microsoft Learn
Thanks and regards,
Anjan Kumar Chippa
- v-achippa
Hi fabricpribeiro,
We wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa