The logical type TimeSpan is not supported in Delta format. Reason: Data Pipeline

[PWTRyanWhite]

Hello,

I've seen this issue pop up for many users. Most responses consist of change the data type at the source, however, I cannot change the data type at the source. I have transformed the destination sink type to string but I keep getting the same issue.

Any other thoughts other than just removing the fields from the data set?

[Anonymous]

Hi @PWTRyanWhite ,

As FabianSchut mentioned, the best way I can think of is to write some query steps as well:

• Custom Conversion: Before writing to the Delta format, you can apply a custom conversion to convert the TimeSpan value to a supported format, such as a string representation of total milliseconds or seconds. This way, you can retain the TimeSpan information without losing data.
• Intermediate storage: Use an intermediate storage format that supports the TimeSpan type, such as Parquet and then convert to Delta format.

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[PWTRyanWhite]

Unfortunately, writing queries is not really an option at my company. Is there another way you can think of? 

[FabianSchut]

Hi @PWTRyanWhite,

Are you able to query your source? You could write a query that uses casting in the select statement to change the column to a type that is supported.