Forum Discussion
Replacing Some data
This might be a stupid question but I can't work out how to crack it.
We have a fairly clunky process for a timesheet Application. We can only get to raw data using exported CSV files. While I can import these quite easily, the problem I have is that the people authorising them can be a little slack in terms of hitting the deadline. The plan is each month we will import all the data from the previous 3 months, to catch up late approvals. Is there any way to have a portion of the existing data deleted based on its date and then bulk load the new data. In the current VBA version, I have the macro delete all records relating to the past 3 months prior to importing the refreshed and new stuff.
I know that I could get it to drop all the data and reimport the lot but going forward that will cause it to have to reimport a load of data which won't have changed
Thanks in advance
You can achieve this with incremental refresh. Either in:
- semantic model: https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
- dataflows: https://learn.microsoft.com/en-us/power-query/dataflows/incremental-refresh
You can remove old data and load the latest records using Power Query in Power BI. You need to:
1. Load existing data
2. Filter out records from the last 3 months
3. Append new data from the CSVYou can also use Incremental Refresh to automate this process.
https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
Let me know if you need further guidance
If this response was helpful, please accept it as a solution and give kudos to support other community members
Hi tonyenglish , There are several ways to achive this through Power Query as well as incrimnetal refresh please try these out
Incremental Refresh:- Set up incremental refresh in Power BI for your dataset.
- Configure it to delete and reload data for the past three months during each refresh.
Filter Old Data:
- In Power Query, filter records older than three months.
- Append the new data to existing data.
If your source is Database then you could use SQL to filter it in the source itself
If this post helped please do give a kudos and accept this as a solution
Thanks In Advance
6 Replies
You can achieve this with incremental refresh. Either in:
- semantic model: https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
- dataflows: https://learn.microsoft.com/en-us/power-query/dataflows/incremental-refresh
You can remove old data and load the latest records using Power Query in Power BI. You need to:
1. Load existing data
2. Filter out records from the last 3 months
3. Append new data from the CSVYou can also use Incremental Refresh to automate this process.
https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
Let me know if you need further guidance
If this response was helpful, please accept it as a solution and give kudos to support other community members
Hi tonyenglish , There are several ways to achive this through Power Query as well as incrimnetal refresh please try these out
Incremental Refresh:- Set up incremental refresh in Power BI for your dataset.
- Configure it to delete and reload data for the past three months during each refresh.
Filter Old Data:
- In Power Query, filter records older than three months.
- Append the new data to existing data.
If your source is Database then you could use SQL to filter it in the source itself
If this post helped please do give a kudos and accept this as a solution
Thanks In AdvanceHi tonyenglish,
Thank you for reaching out in Microsoft Community Forum.
Thank you Akash_Varuna, ArwaAldoud, MarkLaf for the helpful responses.
As suggested by Akash_Varuna, ArwaAdoudOmer, MarKLaf, I hope this information was helpful. Please let me know if you have any further questions or you'd like to discuss this further. If this answers your question, please "Accept as Solution" and give it a 'Kudos' so others can find it easily.
Please continue using Microsoft community forum.
Regards,
Pavan.Hi tonyenglish,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, kindly "Accept as Solution" and give it a 'Kudos' so others can find it easily.
Thank you,
Pavan.Hi tonyenglish,
I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Please continue using Microsoft community forum.
Thank you,
Pavan.
