Forum Discussion
Refresh and duplicate rows
Hello everyone,
I have set up my dataset by brining in multple shared folders. The first shared folder is the base folder (The CSV file soes not change). The second folder contains a CSV file that refreshes weekly and included new data and changed data. This changed data will impact the base data set, and will create duplicate rows. I have transformed and combined both querys in power query.
I want to set up a refresh policy to only refresh new/changed data and elmiate duplicate rows. I have a data gateway connected already so my only issue is how I would do this.
Anyone have any suggestions? I have a unique indenifier in my data set which would sniff out duplicate rwos and I have a last changed date in my data set as well.
Is there any M code, DAX function, Incremental refresh policy parameters that would work for me? The end goal would be to only replace changed data and load new data.
2 Replies
Hi PowerRocky12 ,
You might wanna check out the incremental refresh feature in Power BI:
Configure incremental refresh and real-time data for Power BI datasets - Power BI | Microsoft Docs
Let me know, if this helps 🙂
/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/I don't think a basic incremental refresh is going to help me but could be wrong. Could you please explain and break down for me how it will help with my questions above?
In my data set I have +20 years of historical data that will have changes every weekly data load. I would have to leave the parameters pretty wide open and would defeat the purpose for this feature.
