topic Re: Create historic data table with Web incoming data in Developer

Create historic data table with Web incoming data

Anonymous — Tue, 19 May 2020 10:13:48 GMT

Hello all. I did a lot of research here and so far i couldn't find any suitable solution. I create a table in Power Query Editor that contains sensor data directly from Web. This data is a CSV-dat file and usually updated with new lines every 2 hours. The "current" is the month May in this excercise - see photo attached. In June this file will contain only "June" data and the "May" data will then be available under a different file name. The files are small (<1000 rows) so data handling does not seem to be a issue.

Now i want to create Visual Graphs for this hourly-updated data and publish the report. Is there any way - in every refresh - to collect the newly incoming data into a table ? I want to to automate the process and not having to manually load/join tables every month.

"Incremental refresh" here would not work as the incoming data contain only the current month.

Other commands such as "Union" will also not work.

I hope the above was clear - any input and guidance appreciated! Regards

Re: Create historic data table with Web incoming data

Greg_Deckler — Tue, 19 May 2020 10:25:33 GMT

So generally no. What I would recommend would be to setup a process via Power Automate or other means by which files are copied to a single folder and given a unique name. Then use a Folder query in Power BI.

Re: Create historic data table with Web incoming data

Anonymous — Tue, 19 May 2020 15:27:55 GMT

I see...it's getting really complicated then..Thanks for the answer.

Re: Create historic data table with Web incoming data

Anonymous — Tue, 26 May 2020 07:34:26 GMT

Dear Greg and all,

i finally managed to do it using a Dataflow and a Dataset. The i schedule refresh intervals for both and the report is always up to date.

More information on how to do it you can find under this page - which was really heplful for me :

https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows

So far i don't see any drawbacks with this approach apart from the fact tha the data update is a bit slow.

Antonis

Re: Create historic data table with Web incoming data

Anonymous — Wed, 03 Jun 2020 20:55:08 GMT

Dear Greg,

i finally managed to do it using a combination of dataflows and datasets. The update interval has to be set with a difference of 30 minutes in order for this to work correctly.

Some instructions can be found under this link https://exceed.hr/hr/blog/incremental-refresh-using-powerbi-pro-and-dataflows