Forum Discussion
PowerBi desktop Incremental refresh
To effectively manage datasets with 10m+ rows you will need some kind of database solution, Incremental refresh needs to use query folding. If you stick with csv files it's always going to be slow.
If you go with the local database approach you would just need a process to insert the data into your DB table.
I would have an extra column in the table for the insert date and use that for the incremental refresh process.
If you need a free tool to help you with the ETL then KNIME might be worth looking at https://www.knime.com/
To effectively manage datasets with 10m+ rows you will need some kind of database solution, Incremental refresh needs to use query folding. If you stick with csv files it's always going to be slow.
If you go with the local database approach you would just need a process to insert the data into your DB table.
I would have an extra column in the table for the insert date and use that for the incremental refresh process.
If you need a free tool to help you with the ETL then KNIME might be worth looking at https://www.knime.com/