Unable to properly merge text file using copy activity in Ms fabric.

[Anonymous]

I have two text files in Blob Storage that I’m trying to merge using a Microsoft Fabric pipeline with a copy activity. The merge works, but the first row from one of the files disappears. Has anyone else experienced this issue?

[Anonymous]

Hi @Anonymous 

Some sources and destinations might not have an option to specify whether the files have a header row. In that case, one alternative is to use the Dataflow Gen2 instead of Copy data activity. In dataflow Gen2, you can explicitly set the first row as headers or not before copying the data to the destination data storage. 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

[Anonymous]

But I our files are stored in ADLS container so will Dataflow Gen2 works ??

[Anonymous]

Hi @Anonymous 

Dataflow Gen2 supports the data source Azure Data Lake Storage Gen2. Azure Data Lake Storage Gen2 connector overview

However, for the destination, it doesn't support ADLS. Dataflow Gen2 data destinations and managed settings

Best Regards,
Jing

[lbendlin]

Did you mean to say "append"  rather than "merge" ?

[Anonymous]

yes I want to append two text file but while appending 1st line got vanished.

[lbendlin · Accepted solution]

The process assumes that the files have header rows. During an append the header row of each but the first file will be discarded.

Specify that they files don't have headers, and provide your own column names.