topic Loading data from Lakehouse to Snowflake using Copy data in Data Pipeline in Data Engineering

Loading data from Lakehouse to Snowflake using Copy data in Data Pipeline

shivaazure — Tue, 12 Nov 2024 22:49:13 GMT

I have source Lakehouse table and target snowflake, using datalake copy data

trying to preview the source lakehouse data, but getting this error:

The data type is not supported in Delta format. Reason: Cannot find supported logical type for column name Extended_Status, delta type void

I am writing spark dataframe into lakehouse table. few columns contain nested JSON format.

Re: Loading data from Lakehouse to Snowflake using Copy data in Data Pipeline

Anonymous — Wed, 13 Nov 2024 06:52:07 GMT

Hi @shivaazure ,

The error message indicates that void type is not supported. I have the following suggestions:

Before writing to the Delta table, you can convert the data type of the column in the notebook:

from pyspark.sql.functions import col from pyspark.sql.types import StringType #Convert the column to string type df = df.withColumn("Extended_Status", col("Extended_Status").cast(StringType()))

Then use the following code to create the Delta table:

df.write.mode("overwrite").format("delta").saveAsTable("your_table_name")

Try again to see if the error still occurs.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Loading data from Lakehouse to Snowflake using Copy data in Data Pipeline

Anonymous — Tue, 19 Nov 2024 06:15:02 GMT

Hi @shivaazure ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

Re: Loading data from Lakehouse to Snowflake using Copy data in Data Pipeline

Anonymous — Thu, 28 Nov 2024 09:02:39 GMT

Hi @shivaazure ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team