topic Mirrored Azure Databricks catalog is missing columns in Fabric in Data Engineering

Mirrored Azure Databricks catalog is missing columns in Fabric

FabianSchut — Fri, 31 Jan 2025 16:12:06 GMT

Hi all,

We have a Mirrored Azure Databricks catalog running in Fabric. Today we noticed that there are several columns missing. When we look closer at the columns that are missing, the column type is timestamp_ntz in databricks. We don't know how to get those columns to Fabric. Does anyone have a solution for this?

We already tried to recreate the Mirrored Azure Databricks catalog, without success.

I also found this Microsoft article stating that you should "Select the Configure replication button", but we cannot find this button : https://learn.microsoft.com/en-us/fabric/database/mirrored-database/troubleshooting#columns-are-missing-from-the-destination-table

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

nilendraFabric — Fri, 31 Jan 2025 19:45:44 GMT

Hello @FabianSchut

Seems like `TIMESTAMP_NTZ` is not fully compatible with Fabric’s SQL analytics endpoints.This incompatibility can result in columns being excluded or not displayed properly during schema synchronization.

is this possible for you to convert TIMESTAMP_NTZ` column to a `TIMESTAMP` in Databricks.

When you do this conversion in Databricks

it will interpret the timestamp using the current session time zone, because `TIMESTAMP_NTZ` values do not include time zone information and must be mapped to a particular zone when converted.

Thanks

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

FabianSchut — Sat, 01 Feb 2025 06:47:58 GMT

Thank you for your answer! I was hoping that there would be a solution without converting the data type. I will try this next week and will let you know!

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

v-karpurapud — Mon, 03 Feb 2025 06:14:45 GMT

Hi @FabianSchut

Thanks for reaching out to Microsoft Fabric Community Forum.

Thank you for your response @nilendraFabric ! I appreciate your insights.

Additionally, if you do not prefer to convert the data type, consider the following steps:

Check Permissions:
Ensure that the necessary permissions are granted for the schema in Unity Catalog that contains the tables to be accessed from Fabric. The EXTERNAL USE SCHEMA privilege must be explicitly granted.
Configure Replication:
Verify that the tenant setting "Mirrored Azure Databricks Catalog" is turned on in the Microsoft Fabric Admin Portal.

Furthermore, we would like to provide some reference links that may be useful:

Microsoft Fabric Mirrored Databases From Azure Databricks (Preview) Tutorial - Microsoft Fabric | Microsoft Learn
Microsoft Fabric Mirrored Catalog From Azure Databricks (Preview) - Microsoft Fabric | Microsoft Learn

If my response has resolved your query, please mark it as the Accepted Solution to help others. Additionally, I would appreciate a 'Kudos' if you found my response helpful.

Thank you!

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

v-karpurapud — Thu, 06 Feb 2025 15:46:35 GMT

Hi @FabianSchut

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

v-karpurapud — Mon, 10 Feb 2025 06:56:24 GMT

Hi @FabianSchut

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

v-karpurapud — Fri, 14 Feb 2025 11:12:25 GMT

Hi @FabianSchut

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

Re: Mirrored Azure Databricks catalog is missing columns in Fabric

FabianSchut — Sun, 23 Feb 2025 11:50:32 GMT

I'm sorry for my late reply. I did not have time to work on it lately. I went for the option to cast the data types to string (which was good enough in this case).