Forum Discussion
Mirrored Metadata Catalog for Databricks - No Data Agent?
Hi dbeavon3,
I think there is an important distinction here between a Mirrored Azure Databricks Catalog and a Fabric Mirrored Database.
Microsoft’s current Fabric Data Agent data-source documentation lists Lakehouse, Warehouse, Fabric SQL Database and Mirrored Database as supported SQL sources.
For a Lakehouse specifically, Data Agent queries the Delta tables that are exposed through the Lakehouse SQL analytics endpoint.
Azure Databricks catalog mirroring works a little differently. Microsoft’s mirroring documentation explains that Databricks catalog mirroring synchronizes the Unity Catalog metadata while the underlying data is accessed through OneLake shortcuts.
I do not currently see Mirrored Azure Databricks Catalog listed as a Data Agent source that can be attached directly.
However, Microsoft does document a supported bridge: you can create Lakehouse shortcuts to the mirrored Databricks catalog.
The important part is where those shortcuts are created.
If the Databricks Delta tables are added under the Lakehouse Tables section, Fabric can register them as tables and expose them through the Lakehouse SQL analytics endpoint.
I would test this first:
- Open the Lakehouse SQL analytics endpoint.
- Check whether one of the Databricks-backed shortcut tables appears there.
- Run a simple SELECT against it.
- If that works, try adding that Lakehouse - rather than the mirrored Databricks catalog item itself - as the Data Agent source.
Microsoft’s Lakehouse shortcut documentation notes that shortcuts in the Tables section can be queried through both Spark and the SQL analytics endpoint, while shortcuts under Files are not automatically registered as SQL tables.
So my suspicion would be:
Databricks Unity Catalog
-> mirrored Databricks catalog
-> Lakehouse table shortcut
-> Lakehouse SQL analytics endpoint
-> Fabric Data Agentrather than attaching the mirrored catalog directly to the Data Agent.
If the shortcut table is already visible and queryable from the SQL analytics endpoint but the Data Agent still gives “Couldn't add data source,” then I think you have a much stronger case for a current Data Agent Preview limitation or product issue.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
- dbeavon31 day ago
Memorable Member
Thanks a lot for the reply.
1. Yes, the sql endpoint works fine (eg. via SSMS)
2. Yes shortcuts work. But remember that mirrored unity catalog metadata is also using shortcuts under the hood.
3. Yes select works.
4. This is exactly what I'm hoping to avoid. The benefit of using mirrored UC metadata is that it will keep the entire table intact, and will keep syncing metadata as it is changed over time.
FYI, The blog link I shared is explicitly saying that UC metadata catalogs are a supported scenario
... I have opened a ticket today, and the engineer confirmed that this should work. He is speaking to PG and they say there may be a regression. Our workloads run in the NCUS region and we often see bugs that other regions don't encounter.
Hopefully they will fix this soon. I don't have an ETA for a fix yet. Have a nice weekend.