topic Re: Mirrored Azure Databricks Catalog in Data Engineering

Mirrored Azure Databricks Catalog

Srisakthi — Fri, 27 Sep 2024 09:37:12 GMT

Hi Everyone,

Has anyone tried using the latest feature of Mirroring Azure databricks unity catalog ?. I was able to successfully mirror unity catalog tables, but I'm facing an error with Unmanaged tables. I have tried User identity as well Service Principal. Enabled “External Data Access” on the metastore, “EXTERNAL_USE_SCHEMA” , “SELECT”,“USE” permissions has been provided already.

I can query the managed table but i cant query unmanaged table. ADLS(unmanaged table) is not under any private network .

Please suggest if i have missed any permissions.

Thanks,

Srisakthi

Re: Mirrored Azure Databricks Catalog

Anonymous — Mon, 30 Sep 2024 03:10:41 GMT

Hi @Srisakthi

I haven't used this preview feature yet. I went through the relevant documentations, it seems you’ve already covered many of the necessary steps for setting up and querying unmanaged (external) tables in Azure Databricks Unity Catalog. Here are a few additional things you might want to check:

Storage Credential and External Location: Ensure that the storage credential and external location are correctly configured and associated with the external table. The external location should be accessible with the storage credential you have set up.
Permissions on External Location: Verify that the necessary permissions are granted on the external location. This includes ensuring that the user or service principal has theREAD and WRITE permissions on the Azure Data Lake Storage (ADLS) path.
Schema Permissions: Double-check that the EXTERNAL_USE_SCHEMA, SELECT, and USE permissions are correctly applied to the schema containing the external table.
Network Configuration: Since your ADLS is not under any private network, ensure that there are no network restrictions or firewall rules that might be blocking access to the external storage.
Error Messages: Review the error messages when you failed to query from the external tables. It might give more insight into what is going wrong. Did you see any specific error messages?

If you’ve verified all these aspects and are still facing issues, it might be helpful to reach out to Fabric support for more detailed troubleshooting.

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

Re: Mirrored Azure Databricks Catalog

Srisakthi — Mon, 30 Sep 2024 08:25:01 GMT

Hi Jing, Yeah as I mentioned earlier all those options are enabled already. Here is the screenshot of the error.

Thanks,

Srisakthi

Re: Mirrored Azure Databricks Catalog

Anonymous — Mon, 30 Sep 2024 09:30:13 GMT

Hi @Srisakthi

Have you tried using other tools to connect to its SQL analytics endpoint and query data from the unmanaged tables? Would it succeed?

Best Regards,
Jing

Re: Mirrored Azure Databricks Catalog

Srisakthi — Tue, 01 Oct 2024 06:03:59 GMT

Hi Jing, Those tables are not listing in SQL endpoint.

Re: Mirrored Azure Databricks Catalog

lesommf — Tue, 01 Oct 2024 11:36:46 GMT

Hello,

I have exactly the same problem, i was able to mirror tables, i see the tables in the onelake but impossible to query the tables (external locations) and i dont see them in the sql endpoint of fabric...

Thank you

Re: Mirrored Azure Databricks Catalog

aaron2 — Tue, 01 Oct 2024 16:44:24 GMT

Experiencing the same issue. Complete the catalog mirror connection, but receive same error message as the others and also don't see any tables in the sql endpoint. When clicking 'Go to mirrored database' in the SQL endpoing page, nothing ever loads.

Re: Mirrored Azure Databricks Catalog

aaldo_666 — Thu, 05 Dec 2024 08:17:24 GMT

I have very same issue. All is set up according to documentation and tables look synched with the databricks unity catalog. But I cannot view the data.
Is this related to firewall or VNET?

Re: Mirrored Azure Databricks Catalog

Srisakthi — Fri, 06 Dec 2024 12:16:35 GMT

Hi @aaldo_666 ,

Have you checked your access to the underlying storage.? There could be a possiblity of firewall as well. Please check all the points mentioned above by @Anonymous .

Regards,

Srisakthi

Re: Mirrored Azure Databricks Catalog

Mahens — Fri, 06 Dec 2024 13:40:38 GMT

One way to know underlying issue is , Try creating OneLake shortcut to Databricks artifact shortcut in schema enaled Lakehouse and see error while accessing table data.

Mostly either user who configured databricks connection might not have right permission or underlying storage account is behind firewall.

Re: Mirrored Azure Databricks Catalog

harrychenO365 — Thu, 20 Feb 2025 23:49:37 GMT

Any solution on this? I've the same sisue to mirrowing Databricks.

Thanks.

Re: Mirrored Azure Databricks Catalog

Srisakthi — Fri, 21 Feb 2025 06:59:59 GMT

Hi @harrychenO365 ,

Please follow the solution provided by @Anonymous . It will solve the problem.

Regards,

Srisakthi

Re: Mirrored Azure Databricks Catalog

Mahens — Fri, 21 Feb 2025 13:25:00 GMT

For all the people facing issue in previewing data. Please follow instructions

1. You must enable external data access on the metastore. For more information, see

Enable external data access on the metastore

2. You must have the EXTERNAL USE SCHEMA privilege on the schema in Unity Catalog that contains the tables that will be accessed from Fabric. For more information, see https://learn.microsoft.com/en-us/azure/databricks/external-access/admin#external-schema

Currently we dont support for storage account behind firewall. Please ensure your storage account is enabled with public network access.

Re: Mirrored Azure Databricks Catalog

Mahens — Fri, 21 Feb 2025 13:25:25 GMT

For all the people facing issue in previewing data. Please follow instructions

1. You must enable external data access on the metastore. For more information, see

Enable external data access on the metastore

Currently we dont support for storage account behind firewall. Please ensure your storage account is enabled with public network access.