Streaming Dataset or Dataflow to store historical data ?

[philthegaps]

Hi there

I would like to store historical data on a daily basis to make trend KPI and I wonder which technology is the more relevant. Note that the old data are removed from the source (an SQL database with a custom OLEDB connector) and I don't really see if it can be stored into the dataflow or in the streaming dataset. My data model is not straighforward, and I don't think that simple json structures used in the streaming dataset could fit, so I guess dataflow should be the solution, however in all the examples I saw, the data are not deleted from the source, so I can't figure out what to do with Power BI service. 

[GilbertQ · Accepted solution]

Hi there

If your source data is being deleted you could potentially store it in a dataflow. But if you ever needed to refresh the data you would loose it because the historical data is not there.

What I would suggest is to rather store the data from your original source into another database to keep the history.

And then load from the historical database?

[philthegaps]

Well if you are right, this is indeed the best solution.

[philthegaps]

Well as I can get the data from my sql source, I will create a scheduled task to extract them every week and store them in another database with a new "week number" column. It is the simpler solution.

[GilbertQ]

Thanks for letting us know.