Forum Discussion
Create historic data table with Web incoming data
Hello all. I did a lot of research here and so far i couldn't find any suitable solution. I create a table in Power Query Editor that contains sensor data directly from Web. This data is a CSV-dat file and usually updated with new lines every 2 hours. The "current" is the month May in this excercise - see photo attached. In June this file will contain only "June" data and the "May" data will then be available under a different file name. The files are small (<1000 rows) so data handling does not seem to be a issue.
Now i want to create Visual Graphs for this hourly-updated data and publish the report. Is there any way - in every refresh - to collect the newly incoming data into a table ? I want to to automate the process and not having to manually load/join tables every month.
"Incremental refresh" here would not work as the incoming data contain only the current month.
Other commands such as "Union" will also not work.
I hope the above was clear - any input and guidance appreciated! Regards
Dear Greg and all,
i finally managed to do it using a Dataflow and a Dataset. The i schedule refresh intervals for both and the report is always up to date.
More information on how to do it you can find under this page - which was really heplful for me :
https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows
So far i don't see any drawbacks with this approach apart from the fact tha the data update is a bit slow.
Antonis
Dear Greg,
i finally managed to do it using a combination of dataflows and datasets. The update interval has to be set with a difference of 30 minutes in order for this to work correctly.
Some instructions can be found under this link https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows
4 Replies
So generally no. What I would recommend would be to setup a process via Power Automate or other means by which files are copied to a single folder and given a unique name. Then use a Folder query in Power BI.
Dear Greg and all,
i finally managed to do it using a Dataflow and a Dataset. The i schedule refresh intervals for both and the report is always up to date.
More information on how to do it you can find under this page - which was really heplful for me :
https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows
So far i don't see any drawbacks with this approach apart from the fact tha the data update is a bit slow.
Antonis
Dear Greg,
i finally managed to do it using a combination of dataflows and datasets. The update interval has to be set with a difference of 30 minutes in order for this to work correctly.
Some instructions can be found under this link https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows
