Forum Discussion
Pipeline Copy activity CSV to Parquet issue
I found the issue to be that many rows with null values were intentionally added to the CSV, and after partitioning, the nulls were the first to be seen.
File use comma as delimiter and " as a quote character
Super UserSo commas are quoted. But what about linefeeds in your data? are they quoted?
I found the issue to be that many rows with null values were intentionally added to the CSV, and after partitioning, the nulls were the first to be seen.
Hi A_monged ,
Thanks for the reply from lbendlin .
In order to deal with null values, an efficient way to get the data is to use Dataflow Gen2. In Dataflow Gen2, you can process the data, such as removing columns that contain null values. Then, set Destination to Lakehouse so that data that does not contain null values can be written to Lakehouse. This approach ensures data integrity and accuracy.
For more information on using Dataflow Gen2, you can refer to these official documents:
Create your first Microsoft Fabric dataflow - Microsoft Fabric | Microsoft Learn
If you have any other questions please feel free to contact me.
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
There are no linefeeds in my data