topic Re: Copy data de duplication in Data Engineering

Copy data de duplication

Elisa112 — Tue, 23 Jul 2024 09:44:22 GMT

Hello, I am very new to synapse and am tasked with getting data from cloud azure database to a dedicated pool. So far I have created a copy task to bring the data in, so far so good, the copy task is now populating tables (around 20 tables) , however now the issue is duplication and increasing data. What is the best option for deduplicating the tables, should I now create pipeline tasks to drop the tables each day, since a full data refresh is needed daily and if so how should this be done in the simplest and most efficient way?

All assistance greatly received.

Thank you

Re: Copy data de duplication

Anonymous — Wed, 24 Jul 2024 01:12:51 GMT

HI @Elisa112,

Perhaps you can try to invoke the query editor for further data cleanup in the data pipeline if it suitable for your requirements:

Use a dataflow in a pipeline - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

Re: Copy data de duplication

Element115 — Fri, 04 Apr 2025 22:57:36 GMT

@Elisa112 You need to implement an incremental copy algorithm so you only ingest new data every day instead of ingesting everything from scratch again, day after day.