Getting governed Databricks data into Fabric behind a private network

In a personal lab environment, I evaluated a proof-of-concept architecture for mirroring a Databricks silver layer into Microsoft Fabric and building a gold layer in Fabric on top of it. The silver layer is just the example I used; the same approach works if you would rather mirror gold.

The constraint that shaped the work is that the whole Databricks estate is private. The workspace uses private endpoints, serverless egress is pinned to a Network Connectivity Configuration, and the storage account denies public access. So the real questions were how the data crosses that private boundary, and what it takes to keep Unity Catalog governance in force once the data is in Fabric.

I have compared four ways to get the data across.

The four setups

Governance is the deciding factor between the four. Unity Catalog row and column policies do not travel to Fabric on their own. Where the data lands in OneLake you govern it with OneLake security, and Policy Weaver can sync existing Unity Catalog grants into OneLake roles.

A practical note: if you are only bringing a silver layer across, fine-grained Unity Catalog policies are often not critical, since silver is usually an engineering layer rather than the reporting surface. In that case you can build the access model directly in Fabric with OneLake security and skip the policy translation. Reach for Policy Weaver when you need to carry existing row filters and column masks over as they are.

What we tested

• How to bring a Unity Catalog catalog into Fabric when the workspace and storage sit behind a private network. This is a supported pattern through a VNet data gateway and trusted workspace access, so the goal was the exact steps and the gotchas rather than proving it can be done.
• Whether row filters and column masks still apply for users querying through Fabric, and how to restore them if not.
• Which of the four setups works best today behind a private network.

Environment:

• Azure Databricks (Premium) with Unity Catalog, workspace behind private endpoints.
• A Unity Catalog storage account on Selected networks, public access denied.
• A Microsoft Fabric capacity and workspace.
• Two Entra ID groups federated into Databricks: one admin, one restricted analyst.

Shared groundwork: the private network

All four setups sit on the same storage firewall shape. We kept the Unity Catalog storage account on Selected networks, not fully disabled:

• Set to Selected networks (publicNetworkAccess = Enabled with defaultAction = Deny), not fully disabled.
• Resource instance rules for the Databricks access connector, the Purview scanner, and Microsoft.Fabric/workspaces/*.
• Trusted service exception on (bypass = AzureServices).

Two findings here saved us time later:

• Fully disabling public access breaks Fabric mirroring with a 403, while Databricks serverless keeps working over its private endpoints. That makes the failure look unrelated to storage.
• We tried turning the trusted service exception off (bypass = None). Metadata reads kept working, but the Fabric data-plane read failed to load table data, so bypass = AzureServices has to stay on. Firewall changes take a couple of minutes to propagate, so a 403 right after a change is usually stale cache.

Databricks serverless reaches the storage through the Network Connectivity Configuration with serverless private endpoints for both the blob and dfs subresources, approved on the Azure side.

Setup 1: Mirror Unity Catalog from Fabric (GA)

Zero copy. You create the mirror from inside Fabric, point it at the workspace and catalog, and Fabric keeps a live read-only copy that reads the Delta through Unity Catalog credential vending. You build gold on top with a notebook and/or materialized views.

XOR with Materialized Lake Views: 

Setup, step by step:

1. Connection. Create a VNet data gateway into the Databricks VNet, then a Fabric Azure Databricks connection bound to the mirroring service principal.
2. Read path to storage. Turn on trusted workspace access, so the Fabric workspace identity is trusted by the storage account.
3. Credential vending. Grant the mirroring principal on the catalog: - USE CATALOG - USE SCHEMA - EXTERNAL USE SCHEMA (not part of ALL PRIVILEGES; only the catalog owner can grant it)
4. Create the mirror. From Databricks (Share, Publish to OneLake) or from Fabric (New item, Mirrored Azure Databricks Catalog). The name allows letters, numbers, and underscores only. - 403 on publish: the capacity is not assigned. - 404 on publish: the capacity is assigned but not running.

The governance catch: row filters and column masks do not travel to Fabric, and credential vending cannot read a table that still has one applied. Those tables show as "Couldn't Access Data", so the tables you most want to protect are the ones the mirror cannot read. Policy Weaver closes the gap:

1. Apply the group-based filters and mask in Unity Catalog.
2. Give the Policy Weaver service principal Member on the Fabric workspace.
3. Run it. It produces one OneLake role per group, plus the built-in DefaultReader.
4. Detach the filters and mask so the mirror can read again. The functions stay defined, so it is reversible.

ALTER TABLE silver.activity DROP ROW FILTER;
ALTER TABLE silver.activity ALTER COLUMN cost_amount DROP MASK;

Do not re-run Policy Weaver after detaching. With no policies to find, it does a full replace and wipes the roles. Plain table-level SELECT grants translate directly and need no attach and detach. Keep DefaultReader; it is the baseline read grant, and deleting it leaves zero access. Roles are additive, so a restricted user should reach the data only through the restrictive role, never a broad item share.

We proved it in Power BI with Direct Lake on the mirror over SSO (DirectQuery on the SQL endpoint also works; Import bakes in the author's view). Opening one report as an admin and then as an analyst, the row count dropped, the region breakdown narrowed to one region, and the cost column went from real values to blanks, with no report edits and no Power BI RLS.

Setup 2: Store in OneLake, then surface it (Beta)

This is really just the storage layer. Using the Databricks OneLake external location (Beta), Unity Catalog writes managed tables straight into a Fabric storage item. There is a single copy in OneLake and no gateway on the data path; Databricks needs only outbound access to OneLake, over user delegated SAS enabled per workspace.

Permissions:

• The access connector managed identity gets Contributor or higher on the Fabric workspace that holds the storage item, and is bound to a storage credential in Databricks.
• A tenant admin enables creating Databricks storage items.
• The workspace enables user delegated SAS tokens.

What we found: the write side works, but consuming the data in Fabric is where the Beta shows its age. The storage item exposes only a raw file tree, not tables. There is no SQL endpoint over it, the shortcut picker does not list it, and publishing did not produce a usable mirror. The reliable read today is a Spark notebook that loads the Delta files by their OneLake path. One sharp edge: any non-viewer principal in the workspace can read, write, and delete the storage item regardless of Unity Catalog grants, so a dedicated, locked-down workspace is mandatory.

Setup 3: Store in OneLake, publish the metadata, govern with Policy Weaver (Beta)

This builds on Setup 2. You land the storage the same way, also publish the Unity Catalog so Fabric gets the metadata, then run Policy Weaver to translate the grants and policies into OneLake roles. Together the storage item and the published catalog behave close to Setup 1, assembled by hand.

• Networking: the storage layer is outbound to OneLake as in Setup 2; the published catalog uses the same two hops as Setup 1 (VNet gateway for metadata, trusted workspace access for the read).
• Permissions: the access connector identity from Setup 2, a publish identity with USE CATALOG, USE SCHEMA, EXTERNAL USE SCHEMA, and SELECT, and the Policy Weaver service principal as Member on the workspace.
• What we found: the last mile is missing. You cannot yet wire the storage item into trusted workspace access the way the managed mirror does, so you cannot fully close the loop today. It is also more moving parts to keep in sync than Setup 1.

Setup 4: Shortcut to storage, or copy and ingest (GA)

The fallback when the data can stay in the storage account. A shortcut is zero copy and read only; a copy or ingest makes a writable second copy.

• Networking: trusted workspace access with a managed private endpoint, or a VNet gateway.
• Governance: Unity Catalog is out of the read path, so you govern in Fabric with OneLake security. Unity Catalog policies do not apply to raw file shortcuts, and there is no Unity Catalog lineage on the Fabric side.

Use the shortcut when you want no duplication and can govern in Fabric. Use a copy when you need an independent dataset, and plan a refresh strategy.

What Policy Weaver translates

Policy Weaver is a third part package that brings governace from Unity Catalog to Onelake Security. It reads the live Unity Catalog access model and rebuilds it as OneLake data access roles, so governance defined once in Databricks is enforced in Fabric.

Two format details mattered, or a policy quietly fell back to deny: write the column mask as CASE WHEN is_account_group_member('grp') THEN <col> ELSE <x> END with the function parameter named exactly like the masked column, and use one row-filter function per table whose parameter matches that table's real column. Policy Weaver reads the live function definitions, so the SQL has to run on Databricks first.

Which one to pick

• Production today, zero copy, Unity Catalog as source of truth: Setup 1. Safest default.
• Land the storage layer in OneLake now and surface it later: Setup 2.
• Combine that storage with published metadata to emulate Setup 1: Setup 3.
• Data can stay in the storage account, or it is file based: Setup 4.

Setups 2 and 3 are the direction of travel as the OneLake Beta matures. The missing piece is a read surface, a SQL endpoint or shortcut support over the storage item. Once that lands, they get a lot more compelling.

Takeaway

Mirroring moves the data, not the access rules, and masked or filtered tables are unreadable until you translate the policies into OneLake roles and detach them. Setup 1 is the safe default today, Setups 2 and 3 are promising once the storage item gets a read surface, and Setup 4 is the simple choice when the data can stay where it is. We tested with a silver schema, but the steps apply to any schema.

Try it yourself

The full setup, the manual prerequisites, a name-prompted deploy script, and sample data are in the repository: Hanifff/databricks-uc-to-fabric.