topic Re: The logical type TimeSpan is not supported in Delta format. Reason: Data Pipeline in Pipelines

The logical type TimeSpan is not supported in Delta format. Reason: Data Pipeline

PWTRyanWhite — Thu, 24 Oct 2024 15:48:59 GMT

Hello,

I've seen this issue pop up for many users. Most responses consist of change the data type at the source, however, I cannot change the data type at the source. I have transformed the destination sink type to string but I keep getting the same issue.

Any other thoughts other than just removing the fields from the data set?

Re: The logical type TimeSpan is not supported in Delta format. Reason: Data Pipeline

FabianSchut — Thu, 24 Oct 2024 17:47:27 GMT

Hi PWTRyanWhite,

Are you able to query your source? You could write a query that uses casting in the select statement to change the column to a type that is supported.

Re: The logical type TimeSpan is not supported in Delta format. Reason: Data Pipeline

PWTRyanWhite — Thu, 24 Oct 2024 20:37:39 GMT

Unfortunately, writing queries is not really an option at my company. Is there another way you can think of?

Re: The logical type TimeSpan is not supported in Delta format. Reason: Data Pipeline

Anonymous — Fri, 01 Nov 2024 08:36:38 GMT

Hi PWTRyanWhite ,

As FabianSchut mentioned, the best way I can think of is to write some query steps as well:

Custom Conversion: Before writing to the Delta format, you can apply a custom conversion to convert the TimeSpan value to a supported format, such as a string representation of total milliseconds or seconds. This way, you can retain the TimeSpan information without losing data.
Intermediate storage: Use an intermediate storage format that supports the TimeSpan type, such as Parquet and then convert to Delta format.

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.