Forum Discussion

FabianSchut's avatar
FabianSchut
Solution Sage
1 year ago
Solved

Mirrored Azure Databricks catalog is missing columns in Fabric

Hi all,   We have a Mirrored Azure Databricks catalog running in Fabric. Today we noticed that there are several columns missing. When we look closer at the columns that are missing, the column t...
data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hello FabianSchut 

     

    Seems like `TIMESTAMP_NTZ` is not fully compatible with Fabric’s SQL analytics endpoints.This incompatibility can result in columns being excluded or not displayed properly during schema synchronization.

     


    is this possible for you to convert TIMESTAMP_NTZ` column to a `TIMESTAMP` in Databricks.

     

    When you do this conversion in Databricks 

    it will interpret the timestamp using the current session time zone, because `TIMESTAMP_NTZ` values do not include time zone information and must be mapped to a particular zone when converted.

     

    Thanks 

nilendraFabric's avatar
nilendraFabric
Super User
1 year ago

Hello FabianSchut 

 

Seems like `TIMESTAMP_NTZ` is not fully compatible with Fabric’s SQL analytics endpoints.This incompatibility can result in columns being excluded or not displayed properly during schema synchronization.

 


is this possible for you to convert TIMESTAMP_NTZ` column to a `TIMESTAMP` in Databricks.

 

When you do this conversion in Databricks 

it will interpret the timestamp using the current session time zone, because `TIMESTAMP_NTZ` values do not include time zone information and must be mapped to a particular zone when converted.

 

Thanks 

  • FabianSchut's avatar
    FabianSchut
    Solution Sage
    1 year ago

    Thank you for your answer! I was hoping that there would be a solution without converting the data type. I will try this next week and will let you know!