Forum Discussion
Helper IPowerBi desktop Incremental refresh
To effectively manage datasets with 10m+ rows you will need some kind of database solution, Incremental refresh needs to use query folding. If you stick with csv files it's always going to be slow.
If you go with the local database approach you would just need a process to insert the data into your DB table.
I would have an extra column in the table for the insert date and use that for the incremental refresh process.
If you need a free tool to help you with the ETL then KNIME might be worth looking at https://www.knime.com/
Impactful IndividualYou could possibly set up a database and personal gateway on your local machine (if you have permission to do that).
Then use that DB as a source for the incremental refresh.
You would just need a method to append your new data to the local DB every time there is an update
Helper IThanks for your reply.
another question. If i am able to create each day a new file with just the data of the last day there is no way to append to the historic table on powerbi, checking for duplicate as incremental refresh?
Because actually when i use simple append powerbi start to calculate the appended table each time and being millions and million of rows it takes a lot of time. I would like just to append eache day the new data