On-Prem Copy Data in Data Pipeline Appending instead of Overwriting

[ilseeb]

Hello,

I am using a Copy Data Activity in a Data Pipeline to copy data from an On-Premises Data Gateway to a Fabric Lakehouse.

I have selected the Overwrite option in the destination tab but the data is being appended.

Each time I run it my data gets duplicated.

Does anyone else have this issue? 

[MJ_Taft]

I have noticed duplicates in our data and was wondering if it is possible to view the ticket or if the support team puts out their findings somewhere

[v-cboorla-msft]

Hi @ilseeb 

Thanks for using Microsoft Fabric Community.

As I understand that you are encountering an issue with the Copy Data Activity in your Data Pipeline within Microsoft Fabric. You’ve selected the Overwrite option in the destination tab, but the data is still being appended, resulting in duplicated data each time you run the pipeline.

Here are some additional troubleshooting steps that could be helpful in resolving the data duplication issue:

Schema Mismatch: Confirm that the schema of the target table in your Fabric Lakehouse matches the schema of the source data. If there’s a discrepancy, the data may be appended instead of overwritten.

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to Microsoft Fabric might be causing the issue. Clearing the cache removes this potentially problematic data.

Hard Refresh: A hard refresh bypasses the cached version of the webpage and forces the browser to download the latest version from the server. Press Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac).

If the issue still persists, please do let us know. Glad to help.

I hope this information helps.

Thank you

[v-cboorla-msft]

Hi @ilseeb 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[ilseeb]

Hello,

I've tried clearing the cookies and caches.

I alos created a new table in the lakehouse and made sure the the shcema is the same but I am still getting duplicates.

[v-cboorla-msft]

Hi @ilseeb 

Apologies for the inconvenience and the delay in response.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[ilseeb]

Thank you, here is the ticket number: 2406140050002255

[jitheshpc1980]

Hi 

I have a similar issue.

I am trying to copy data from a Fabric Wharehouse to another using Copy Data Activity. 

It appends the records instead of overwriting.  

Please help its critical for us.

[jpelham]

We were also experiencing this issue. Typing Overwrite in the table action worked at first, but after some fabric updates it stopped working for us. The alternative solution we got from Microsoft was to use dynamic content in the Table Action input: 

@string('OverwriteSchema')

This has been working well for us now. 

[jitheshpc1980]

There is no Table Action in Copy Data when it comes to WH to WH transfer

[jpelham]

Ah, ok. Our use case is for transfers to lakehouses. I don't have a lot of experience with warehouses but I wonder if you could use a pre-copy script to wipe the old data before loading the new data? 

[jitheshpc1980]

Hi @jpelham , 

thanks pre-copy script worked,  used TRUNCATE statement.

[ilseeb]

Hello,

The conclusion was that something was wrong on my query. I fixed my query and also started a new pipeline from scratch and it has not happened ever since.

If you give more details maybe I can help you. I recently had another issue in which the piepline was running with a succeed status but the tables were not being updated at all.

Someone suggested to use a Dynamic value of Overwrite for the Destination Action instead of selecting it:

I tried this and it has been working. More details on: https://community.fabric.microsoft.com/t5/Data-Pipeline/Lakehosue-table-has-an-error-An-internal-err...

[v-cboorla-msft]

Hi @ilseeb 

Thank you for sharing the details.