Forum Discussion

fabricpribeiro's avatar
fabricpribeiro
Icon for Post Patron rankPost Patron
1 month ago
Solved

How to integrate Databricks into Fabric

Dears,

 

If I am not wrong, there are two options avaialable which we can use to integrate databricks into fabric without having to build costum pipelines 

 

1 - Mirror the Azure Databricks catalog (figure1) 

2 - Create links (but I am not sure about this one)

 

When shall I use Fabric mirrrong Azure Databricks catalog  and when should I use links or shorcuts? Can someone please help understand?  also, are both solutions managed entirely by Fabric?

 

Thanks,

 

Pedro  

 

Figure 1)

 

admin
workspaces
  • v-achippa's avatar
    v-achippa
    1 month ago

    Hi fabricpribeiro,

     

    Yes, Azure Databricks stores delta tables in cloud storage for example ADLS Gen2, while databricks provides the compute and governance layer over that data.

    • If you create a shortcut to a delta table in supported storage, it will appear under the Tables section of your fabric lakehouse. Since the data is stored in delta format, the lakehouse automatically recognizes it as a table, synchronizes the schema and we can easily query it.
    • If you create the shortcut under Files, it remains as raw files and will not automatically be recognized as a table, they will appear under the Files section.

    For permissions, there are two aspects:

    • The shortcut connection must have permission to access the underlying storage where the data resides, this is configured through the shortcut’s connection
    • Once the shortcut is created, users only need the appropriate permissions in fabric to access it. They don't need the databricks permissions because the shortcut reads directly from ADLS Gen2 through the configured connection.
    • If you are using Mirroring for azure databricks(Unity Catalog) instead of an ADLS shortcut, that is different. Mirroring interacts with unity catalog and follows its permissions and supported scenarios.

    Regarding data freshness, shortcuts do not copy the data, they read it in place. As changes are written to the underlying delta tables, fabric reads the latest data from the source, so there isn't a separate synchronization schedule like there is with mirroring.

     

    Thanks and regards,

    Anjan Kumar Chippa

  • v-achippa's avatar
    v-achippa
    1 month ago

    Hi fabricpribeiro,

     

    • Use Shortcuts when you want to access the data where it already exists, avoid data duplication and always work with the latest data in the source storage.
    • Use Mirroring when you want fabric to maintain a managed copy of the supported source data in OneLake so that it can be used across fabric workloads without building custom ingestion pipelines.

    For your reference, follow this below documentation:

    Microsoft Fabric Mirrored Catalog From Azure Databricks - Microsoft Fabric | Microsoft Learn

    Unify Data Sources With OneLake Shortcuts - Microsoft Fabric | Microsoft Learn

     

     

    Thanks and regards,

    Anjan Kumar Chippa

     

12 Replies