Breaking Barriers: Azure Databricks Unity Catalog Tables Can Now Be Stored Directly in OneLake

[Srisakthi]

Background

One of the most common questions customers ask is: How can Azure Databricks and Microsoft Fabric work together without duplicating data? Many teams want a seamless way to read from and write to OneLake while continuing to use Azure Databricks for engineering, analytics, and AI workloads.

With recent interoperability updates, this is now becoming much easier. Azure Databricks can connect to OneLake through Unity Catalog external locations, enabling governed access to OneLake-backed storage. This feature is currently in preview.

How Azure Databricks and Microsoft Fabric Work Together

There are multiple ways to integrate Azure Databricks with Microsoft Fabric depending on whether the goal is to read data, write data, or make Databricks-managed data available in Fabric.

1. Read OneLake Data from Azure Databricks

• Azure Databricks can read OneLake data using the OneLake path, similar to how teams access ADLS Gen2 paths.
• With OneLake catalog federation, Azure Databricks can query OneLake data more easily without creating additional copies.

2. Write Data to OneLake

• Azure Databricks can write to OneLake by using the appropriate OneLake path.
• Mounting also can be used to write to onelake

3. Use Mirroring for Fabric Consumption

• Mirroring makes Azure Databricks data readily available in Microsoft Fabric for downstream consumption.
• This helps Fabric users access Databricks-managed data without building separate movement pipelines.

“Much Awaited Feature”

The latest approach makes the read/write experience even more straightforward. By using Unity Catalog with OneLake external locations, teams can manage access centrally in Azure Databricks while storing data directly in OneLake. This reduces duplication, simplifies governance, and improves collaboration between Databricks and Fabric users.

Getting Started Made Simple

Prerequisites

Azure Databricks

• Azure Databricks workspace enabled for Unity Catalog.
• Databricks Runtime 18.1 or higher, or serverless compute.
• Required privileges on the Unity Catalog metastore: CREATE STORAGE CREDENTIAL and CREATE EXTERNAL LOCATION.
• Azure Managed Identity or Azure Service Principal for authentication.
• Enable External Location for OneLake in Previews tab.

Microsoft Fabric

• Enable Users can create Azure Databricks Storage items in the Fabric Administrator Portal under Tenant settings.
• Enable Authenticate with OneLake user-delegated SAS tokens in the target workspace settings.
• Use a dedicated Fabric workspace for Azure Databricks Storage items where possible, and restrict workspace roles to only the required users or identities.

Step-by-Step Implementation

Step 1: Enable External Location Support for OneLake in Azure Databricks

1. Open Azure Databricks.
2. Select the user icon in the top-right corner and choose Previews.
3. Search for External location and enable the OneLake external location feature.

1. In the search box type, External location and enable the feature

Step 2: Configure Authentication Using Managed Identity or Service Principal

In this example, an existing Access Connector for Azure Databricks with Managed Identity is used for authentication.

1. Create or identify an Access Connector for Azure Databricks with Managed Identity.
2. In Microsoft Fabric, create a new workspace or use a dedicated workspace for this integration.
3. Attach & grant access to the Managed Identity on the Fabric Workspace.

Step 3: Create an Azure Databricks Storage Item in Fabric

1. In the Fabric workspace, create an Azure Databricks Storage item.
2. Ensure the required tenant and workspace settings from the prerequisites section are enabled.
3. After the item is created, capture the workspace ID and Databricks Storage item ID from the URL.

Use the following path format when creating the external location in Azure Databricks:
abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/

Step 4: Create a Storage Credential in Unity Catalog

1. Create a storage credential in Unity Catalog using the Managed Identity configured earlier.
2. Provide the required details, including the Access Connector resource ID (which can be fetched from the Azure portal).
3. Validate that the identity has the necessary permissions before proceeding to external location creation.

Step 5: Create an External Location in Azure Databricks

Once the storage credential is ready, create an external location that points to the OneLake path created in Fabric.

1. Provide a meaningful name for the external location.
2. Choose OneLake as the storage type.
3. Paste the OneLake URL formed in Step 3.
4. Click on Create. If you see a “skipped file events read” warning during the connection test, this is expected behavior and can be ignored.

Step 6: Create the Catalog and Table

With the external location configured, you can now create a catalog and external table that point to OneLake. Replace the workspace ID and Databricks Storage item ID with the values from the created item or from step3.

After the table is created in Azure Databricks, the data becomes available through the configured OneLake storage location in Microsoft Fabric.

Final Thoughts

With OneLake external location support in Unity Catalog, organizations can simplify their lakehouse architecture, reduce unnecessary data movement, and provide governed access across both platforms.

This feature is still in preview, so it is not recommended for production workloads.

References

https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/external-loca...

https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed...