Forum Discussion
Solved
Data Load and coding
Hello, I am looking for some guideance on creating some code to load into my Power BI Data Set. I am trying to write some code simular to how I used it in Qlik. Basically I have two data folders....
You can deduplicate in Power Query, just make sure to wrap your sorted table that puts the most recent results on top within a Table.Buffer, as explained here:
https://exceleratorbi.com.au/remove-duplicates-keep-last-record-power-query/
If you want to partition historical vs. new then you'll want to create two dataflows, it's more efficient than trying to do it all within the dataset. Just to the append/dedupe logic in the dataset (or you could do it in a third dataflow if you don't mind paying for Power BI Premium per User).
Just create two dataflows by folder (historical and new), you don't need to create one for each file (that wouldn't scale at all!). Yes, once you load them in the dataset you can add filtering there if needed.
