Forum Discussion
Hourly Updates multiple files from a folder -
I'm not sure what to do here?
1) I'm reporting against Server Utilization
2) Each Hour there are 7 CSV files dropped into a folder
3) I pull all files from the folder and merge them together so it's one large file and I repor on the data.
The data issue i just realized is that when the new reports come in, they are suppose to be for actvity between 4pm - 5pm, but the file always shows the prior time data as well. So it actually shows all the data points from 1am all the way up to 5pm. Then when the 6pm file comes in, it has all the data from 1am to 6pm.
When i report on the data now, it's full of duplicates from the older files.
Hi sharpedogs
You could add a step "filtered rows",
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each DateTime.Date([datetime])=DateTime.Date(DateTime.LocalNow()) and Time.Hour([datetime])<= Time.Hour(DateTime.LocalNow()) and Time.Hour([datetime])>= Time.Hour(DateTime.LocalNow())-1)Best Regards
MaggieCommunity Support Team _ Maggie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
3 Replies
So, not 100% clear. Are you saying that the 7 files get replaced each hour or get added to the folder?
Regardless, you should be able to filter based upon the time. You could do this dynamically by using a measure that grabs the MAX timestamp in the table. Then you just use that to include everything that has an hour of that measure - 1 hour for example. Would need to better understand your data adn report though. Please see this post regarding How to Get Your Question Answered Quickly: https://community.powerbi.com/t5/Community-Blog/How-to-Get-Your-Question-Answered-Quickly/ba-p/38490
Hi sharpedogs
You could add a step "filtered rows",
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each DateTime.Date([datetime])=DateTime.Date(DateTime.LocalNow()) and Time.Hour([datetime])<= Time.Hour(DateTime.LocalNow()) and Time.Hour([datetime])>= Time.Hour(DateTime.LocalNow())-1)Best Regards
MaggieCommunity Support Team _ Maggie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.Just grab the latest file with each refresh. Make sure you use the Folder import option, then the Combine opreation, which it sounds like you are doing. That will prevent the import from looking at a specific file name.
Set this filter though right before the combine operation starts:
