topic Re: Unable to properly merge text file using copy activity in Ms fabric. in Data Engineering

Unable to properly merge text file using copy activity in Ms fabric.

Anonymous — Fri, 20 Dec 2024 13:39:48 GMT

I have two text files in Blob Storage that I’m trying to merge using a Microsoft Fabric pipeline with a copy activity. The merge works, but the first row from one of the files disappears. Has anyone else experienced this issue?

Re: Unable to properly merge text file using copy activity in Ms fabric.

lbendlin — Fri, 20 Dec 2024 16:51:47 GMT

Did you mean to say "append" rather than "merge" ?

Re: Unable to properly merge text file using copy activity in Ms fabric.

Anonymous — Sat, 21 Dec 2024 05:14:30 GMT

yes I want to append two text file but while appending 1st line got vanished.

Re: Unable to properly merge text file using copy activity in Ms fabric.

lbendlin — Sat, 21 Dec 2024 13:57:26 GMT

The process assumes that the files have header rows. During an append the header row of each but the first file will be discarded.

Specify that they files don't have headers, and provide your own column names.

Re: Unable to properly merge text file using copy activity in Ms fabric.

Anonymous — Mon, 23 Dec 2024 03:26:17 GMT

Hi @Anonymous

Some sources and destinations might not have an option to specify whether the files have a header row. In that case, one alternative is to use the Dataflow Gen2 instead of Copy data activity. In dataflow Gen2, you can explicitly set the first row as headers or not before copying the data to the destination data storage.

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

Re: Unable to properly merge text file using copy activity in Ms fabric.

Anonymous — Mon, 23 Dec 2024 10:31:27 GMT

But I our files are stored in ADLS container so will Dataflow Gen2 works ??

Re: Unable to properly merge text file using copy activity in Ms fabric.

Anonymous — Tue, 24 Dec 2024 01:31:58 GMT

Hi @Anonymous

Dataflow Gen2 supports the data source Azure Data Lake Storage Gen2. Azure Data Lake Storage Gen2 connector overview

However, for the destination, it doesn't support ADLS. Dataflow Gen2 data destinations and managed settings

Best Regards,
Jing