Forum Discussion
Solved
Copy data de duplication
Hello, I am very new to synapse and am tasked with getting data from cloud azure database to a dedicated pool. So far I have created a copy task to bring the data in, so far so good, the copy task i...
HI Elisa112,
Perhaps you can try to invoke the query editor for further data cleanup in the data pipeline if it suitable for your requirements:
Use a dataflow in a pipeline - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng
Elisa112 You need to implement an incremental copy algorithm so you only ingest new data every day instead of ingesting everything from scratch again, day after day.