Pipeline Copy activity CSV to Parquet issue

Hi A_monged ,

Thanks for the reply from lbendlin .

I used PySpark statements in notebook to convert CSV files from lakehouse to parquet files:

# start SparkSession
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CSV_to_Parquet").getOrCreate()

# load CSV file
df = spark.read.format("csv").option("header","true").load("Files/orders/2019.csv")
df.show()

# transform to Parquet form and save it
parquet_file_path = "Files/test.parquet"
df.write.parquet(parquet_file_path)

Works fine, as shown below, and my file does not contain null values:

The error may occur because the data types and schemas defined in the CSV file do not match the data types and schemas expected by the Parquet format.

Optionally, you can use the same method as I did to convert the CSV file to a Parquet file.

If you want to save it as a table you can use the following syntax:

df.write.mode(“overwrite”).saveAsTable(“parquetTestTable”)

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!