Mirrored Azure Databricks Catalog

[Srisakthi]

Hi Everyone,

Has anyone tried using the latest feature of Mirroring Azure databricks unity catalog ?. I was able to successfully mirror unity catalog tables, but I'm facing an error with Unmanaged tables. I have tried User identity as well Service Principal. Enabled “External Data Access” on the metastore, “EXTERNAL_USE_SCHEMA” , “SELECT”,“USE” permissions has been provided already.

I can query the managed table but i cant query unmanaged table. ADLS(unmanaged table) is not under any private network .

Please suggest if i have missed any permissions.

Thanks,

Srisakthi

[Mahens]

For all the people facing issue in previewing data. Please follow instructions

1. You must enable external data access on the metastore. For more information, see  

Enable external data access on the metastore 

2. You must have the EXTERNAL USE SCHEMA privilege on the schema in Unity Catalog that contains the tables that will be accessed from Fabric. For more information, see https://learn.microsoft.com/en-us/azure/databricks/external-access/admin#external-schema

Currently we dont support for storage account behind firewall. Please ensure your storage account is enabled with public network access.

[aaron2]

Experiencing the same issue. Complete the catalog mirror connection, but receive same error message as the others and also don't see any tables in the sql endpoint. When clicking 'Go to mirrored database' in the SQL endpoing page, nothing ever loads.

[v-jingzhan-msft · Accepted solution]

Hi @Srisakthi 

I haven't used this preview feature yet. I went through the relevant documentations, it seems you’ve already covered many of the necessary steps for setting up and querying unmanaged (external) tables in Azure Databricks Unity Catalog. Here are a few additional things you might want to check:

1. Storage Credential and External Location: Ensure that the storage credential and external location are correctly configured and associated with the external table. The external location should be accessible with the storage credential you have set up.

2. Permissions on External Location: Verify that the necessary permissions are granted on the external location. This includes ensuring that the user or service principal has theREAD and WRITE permissions on the Azure Data Lake Storage (ADLS) path. 

3. Schema Permissions: Double-check that the EXTERNAL_USE_SCHEMA, SELECT, and USE permissions are correctly applied to the schema containing the external table.

4. Network Configuration: Since your ADLS is not under any private network, ensure that there are no network restrictions or firewall rules that might be blocking access to the external storage.

5. Error Messages: Review the error messages when you failed to query from the external tables. It might give more insight into what is going wrong. Did you see any specific error messages? 

If you’ve verified all these aspects and are still facing issues, it might be helpful to reach out to Fabric support for more detailed troubleshooting. 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

[Srisakthi]

Hi Jing, Yeah as I mentioned earlier all those options are enabled already. Here is the screenshot of the error.

Thanks,

Srisakthi

[v-jingzhan-msft]

Hi @Srisakthi 

Have you tried using other tools to connect to its SQL analytics endpoint and query data from the unmanaged tables? Would it succeed? 

Best Regards,
Jing

[Srisakthi]

Hi Jing, Those tables are not listing in SQL endpoint.

[aaldo_666]

I have very same issue. All is set up according to documentation and tables look synched with the databricks unity catalog. But I cannot view the data.
Is this related to firewall or VNET?

[Srisakthi]

Hi @aaldo_666 ,

Have you checked your access to the underlying storage.? There could be a possiblity of firewall as well. Please check all the points mentioned above by @v-jingzhan-msft .

Regards,

Srisakthi

[Mahens]

One way to know underlying issue is , Try creating OneLake shortcut to Databricks artifact shortcut in schema enaled Lakehouse and see error while accessing table data.

Mostly either user who configured databricks connection might not have right permission or underlying storage account is behind firewall.

[lesommf]

Hello,

I have exactly the same problem, i was able to mirror tables, i see the tables in the onelake but impossible to query the tables (external locations) and i dont see them in the sql endpoint of fabric...

Thank you

[harrychenO365]

Any solution on this? I've the same sisue to mirrowing Databricks.

Thanks.

[Mahens]

For all the people facing issue in previewing data. Please follow instructions

1. You must enable external data access on the metastore. For more information, see  

Enable external data access on the metastore 

2. You must have the EXTERNAL USE SCHEMA privilege on the schema in Unity Catalog that contains the tables that will be accessed from Fabric. For more information, see https://learn.microsoft.com/en-us/azure/databricks/external-access/admin#external-schema

Currently we dont support for storage account behind firewall. Please ensure your storage account is enabled with public network access.

[Srisakthi]

Hi @harrychenO365 ,

Please follow the solution provided by @v-jingzhan-msft . It will solve the problem.

Regards,

Srisakthi