topic Data Type Conversion in Data Engineering

Data Type Conversion

Kanika123 — Sun, 13 Oct 2024 06:20:13 GMT

I have a pipeline in fabric and source has 2-3 fields that has datetime as datattype and in pipeline I am using copy activity to copy data from source to lakehouse table. Now lakehouse table has fields datatype as timestamp.pipeline is executing successfully but when I am trying to run select query through spark sql on notbook it is returning error datetime inconsistency. Please help me how I can fix this issue

Re: Data Type Conversion

tharunkumarRTK — Sun, 13 Oct 2024 14:15:50 GMT

@Kanika123

I am assuming you are able to view the data from lakehosue explorer and you are facing this error when ran the query in spark sql, is it?

Are you able run a T-SQL query on this table from sql analytics end point?

if the answer for the above two questions is yes then I am guessing its a type conversion problem. Please checkout this blog

https://www.databricks.com/blog/2020/07/22/a-comprehensive-look-at-dates-and-timestamps-in-apache-spark-3-0.html

was there any unsual datetime value like '31-12-9999' ?

I would suggest you to performing manual type casting in the spark sql

BTW, in Fabric lakehouse timestamp datatype is datetime2.

https://learn.microsoft.com/en-us/fabric/data-warehouse/data-types

Need a Power BI Consultation? Hire me on Upwork

Connect on LinkedIn

Did I answer your question? Mark my post as a solution!
If I helped you, click on the Thumbs Up to give Kudos. Proud to be a Super User!

Re: Data Type Conversion

Anonymous — Thu, 24 Oct 2024 09:23:41 GMT

Hi @Kanika123

Is there any update? Have you resolved this issue? If any of the answers provided were helpful, please consider accepting them as a solution. If you have found other solutions, we would greatly appreciate it if you could share them with us. Thank you!

Best Regards,
Jing