Forum Discussion
Using files uploaded daily to update master file
Hello,
I am creating a report that tracks workers' daily hours spent per department using a master file of historical data and a daily uploaded file. The historical data goes from January 2024 to June 2024, and every day a new file will be dropped into a folder that Power Bi is looking at to update what the current "daily uploaded file" is. The daily uploaded file actually includes the past 14 days prior to its upload, of which will often need to overwrite what is in the master file.
I know that I can search a folder for the latest file to use as my "daily upload" source. My concern is that Power Bi won't save data that was gained from previous day's daily upload file, and any days between the last "historical data source" day and the 14th day of the newest daily upload will be lost.
An example of this would be the daily upload from 7/17/24 will contain data from 7/3 to 7/16, and if the historical data only goes to 6/30/24 I'm missing 2 days of data.
Is this something that is possible in Power Bi?
6 Replies
Since 'Remove Duplicates' keeps the record that was initially loaded, wouldn't that prevent the records that are updating from subsequent loads?
- Ritaf1983
Super User
Hi gbpbi
If I understand your question correctly, you're concerned about ending up with only one row per day. That's fine. You can simply create a concatenated column of employee ID + date (or date and time if there are multiple reports per day for each employee) and then remove duplicates based on the concatenated column.
This is an example of what I'm working with. The noteable part is that "File 2" will be uploading later than "File 1," but the data in File 2 should replace any duplicate records from previous loaded files. So in this example, I need Terry's 7/6/2024 Prod 1 hours updated to 2, but if I use 'Remove Duplicates' it is going to remove the "File 2" record because it was loaded after "File 1."
The screenshot is not the files that we are working with, it is just an example of the scenario.
