Forum Discussion

dbeavon3's avatar
dbeavon3
Icon for Memorable Member rankMemorable Member
9 days ago
Solved

Mirrored Metadata Catalog for Databricks - No Data Agent?

Has anyone tested Data Agents in Fabric (NL2SQL)?  I'm connecting to a mirrored lakehouse that uses shortcuts to reach data in ADLS.  The Data agents rely on the SQL endpoint, and related lakehouse t...
lakehouse
  • dbeavon3's avatar
    dbeavon3
    9 days ago

    Thanks a lot for the reply.
    1. Yes, the sql endpoint works fine (eg. via SSMS)
    2. Yes shortcuts work.  But remember that mirrored unity catalog metadata is also using shortcuts under the hood.
    3. Yes select works.
    4. This is exactly what I'm hoping to avoid.  The benefit of using mirrored UC metadata is that it will keep the entire table intact, and will keep syncing metadata as it is changed over time.

    FYI, The blog link I shared is explicitly saying that UC metadata catalogs are a supported scenario



    ... I have opened a ticket today, and the engineer confirmed that this should work.  He is speaking to PG and they say there may be a regression.  Our workloads run in the NCUS region and we often see bugs that other regions don't encounter.  

    Hopefully they will fix this soon.  I don't have an ETA for a fix yet. Have a nice weekend.

ShivekMaharaj's avatar
ShivekMaharaj
Icon for Community Champion rankCommunity Champion
9 days ago

Hi dbeavon3​,

I think there is an important distinction here between a Mirrored Azure Databricks Catalog and a Fabric Mirrored Database.

Microsoft’s current Fabric Data Agent data-source documentation lists Lakehouse, Warehouse, Fabric SQL Database and Mirrored Database as supported SQL sources.

For a Lakehouse specifically, Data Agent queries the Delta tables that are exposed through the Lakehouse SQL analytics endpoint.

Azure Databricks catalog mirroring works a little differently. Microsoft’s mirroring documentation explains that Databricks catalog mirroring synchronizes the Unity Catalog metadata while the underlying data is accessed through OneLake shortcuts.

I do not currently see Mirrored Azure Databricks Catalog listed as a Data Agent source that can be attached directly.

However, Microsoft does document a supported bridge: you can create Lakehouse shortcuts to the mirrored Databricks catalog.

The important part is where those shortcuts are created.

If the Databricks Delta tables are added under the Lakehouse Tables section, Fabric can register them as tables and expose them through the Lakehouse SQL analytics endpoint.

I would test this first:

  1. Open the Lakehouse SQL analytics endpoint.
  2. Check whether one of the Databricks-backed shortcut tables appears there.
  3. Run a simple SELECT against it.
  4. If that works, try adding that Lakehouse - rather than the mirrored Databricks catalog item itself - as the Data Agent source.


Microsoft’s Lakehouse shortcut documentation notes that shortcuts in the Tables section can be queried through both Spark and the SQL analytics endpoint, while shortcuts under Files are not automatically registered as SQL tables.

So my suspicion would be:

Databricks Unity Catalog
-> mirrored Databricks catalog
-> Lakehouse table shortcut
-> Lakehouse SQL analytics endpoint
-> Fabric Data Agent

rather than attaching the mirrored catalog directly to the Data Agent.

If the shortcut table is already visible and queryable from the SQL analytics endpoint but the Data Agent still gives “Couldn't add data source,” then I think you have a much stronger case for a current Data Agent Preview limitation or product issue.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

dbeavon3's avatar
dbeavon3
Icon for Memorable Member rankMemorable Member
9 days ago

Thanks a lot for the reply.
1. Yes, the sql endpoint works fine (eg. via SSMS)
2. Yes shortcuts work.  But remember that mirrored unity catalog metadata is also using shortcuts under the hood.
3. Yes select works.
4. This is exactly what I'm hoping to avoid.  The benefit of using mirrored UC metadata is that it will keep the entire table intact, and will keep syncing metadata as it is changed over time.

FYI, The blog link I shared is explicitly saying that UC metadata catalogs are a supported scenario



... I have opened a ticket today, and the engineer confirmed that this should work.  He is speaking to PG and they say there may be a regression.  Our workloads run in the NCUS region and we often see bugs that other regions don't encounter.  

Hopefully they will fix this soon.  I don't have an ETA for a fix yet. Have a nice weekend.

  • v-abhinavmu's avatar
    v-abhinavmu
    Icon for Community Support rankCommunity Support
    6 days ago

    Hi dbeavon3​,

    Thanks for the update and for sharing the details from your Support investigation.

    Since Microsoft Support has confirmed that this scenario should work and is investigating the issue with the Product Group, working with Support would be the best next step.

    If the issue is resolved, please consider updating this thread as well. It would be helpful for others facing a similar issue.

    Thanks again for keeping the community updated.