Forum Discussion
Incremental Refreshing
Yes you can use incremental refresh in your case. In PQ, go to the table with your data and filter the Date Modified column.
Instead of manually selecting dates, you should be able to apply a filter using the RangeStart and RangeEnd parameters. Then choose Custom Filter and set it to filter for records where Date Modified is greater than or equal to RangeStart and less than RangeEnd.After setting the filter in Power Query, close and apply the changes.
Then, set up the incremental refresh for your 2024 data. Define the period for incremental refresh (refresh the last 1 year) and specify how long you want to keep historical data (keep data for 3 years).You can configure how much data to refresh and whether to archive data. For example, you could set it to refresh the data for the past 1 year (which would cover the changing data for 2024) while keeping the historical data for 2022-2023 unchanged.
If you want to be more specific and only refresh the most recent data, you could set this to 1 month if you only expect data changes for the most recent month.
For example, if you update your data on the 1st of each month, setting this to 1 month will ensure that only the last 30 days of 2024 are refreshed, making the refresh process faster. However, this assumes that older data in 2024 does not change.
Update :
The solution was to convert the column to Datetime
Yes you can use incremental refresh in your case. In PQ, go to the table with your data and filter the Date Modified column.
Instead of manually selecting dates, you should be able to apply a filter using the RangeStart and RangeEnd parameters. Then choose Custom Filter and set it to filter for records where Date Modified is greater than or equal to RangeStart and less than RangeEnd.
After setting the filter in Power Query, close and apply the changes.
Then, set up the incremental refresh for your 2024 data. Define the period for incremental refresh (refresh the last 1 year) and specify how long you want to keep historical data (keep data for 3 years).
You can configure how much data to refresh and whether to archive data. For example, you could set it to refresh the data for the past 1 year (which would cover the changing data for 2024) while keeping the historical data for 2022-2023 unchanged.
If you want to be more specific and only refresh the most recent data, you could set this to 1 month if you only expect data changes for the most recent month.
For example, if you update your data on the 1st of each month, setting this to 1 month will ensure that only the last 30 days of 2024 are refreshed, making the refresh process faster. However, this assumes that older data in 2024 does not change.
Update :
The solution was to convert the column to Datetime
Hi Amira
Thanks so much for your quick reply.
I really can't select my parameter. Its's defined under Other Queries.
But when I go to my table to filter on the date modified colum and select user defined filter it only shows me the dates in my modified colum and not the option for parameters
I'm not sure what I'm doing wrong
Can you explain this part :
I really can't select my parameter. Its's defined under Other Queries.
It's just in a folder called Other queries.
They way that I have my data set up is that I have in the folder Other Queries, one table which imports and combines the data, and then under the folder Data Model I have the Table with the actual data, which is directly linked to the table which imports and combines the data.