Forum Discussion

Yggdrasill's avatar
Yggdrasill
Responsive Resident
2 years ago
Solved

Remove duplicate values in Fabric Data Warehouse SQL table using Stored Procedure

After creating a Pipeline in Data Factory with success where I fetch data via REST API to Azure SQL Database I wanted to see if I could do the same within Microsoft Fabric and use the new (Synapse) D...
Pipelines
Show and Tell
Tutorial Requests
  • Yggdrasill's avatar
    Yggdrasill
    2 years ago

    This SP would work if I want to keep all unique rows but in my case I just want to remove duplicates on my key column [id] but with the highest value in [LastSyncedDate] 

    Your method will still return duplicate id's

    I "solved" this by creating a view in the data warehouse which removes the duplicates and then I just removed the last step of the pipeline and I query the view instead of the table...

Yggdrasill's avatar
Yggdrasill
Responsive Resident
2 years ago

This SP would work if I want to keep all unique rows but in my case I just want to remove duplicates on my key column [id] but with the highest value in [LastSyncedDate] 

Your method will still return duplicate id's

I "solved" this by creating a view in the data warehouse which removes the duplicates and then I just removed the last step of the pipeline and I query the view instead of the table...