PowerQuery/Automate Create Snapshot of DataSet and store within PBI as Calculated Table

[icassiem]

Good day,

I have a requirement to variance of previous vs latest, this worked when source was a 1gb csv file

We have an API source to client that only returns latest and client has no data store to store data.

I am thinking of way to automate a solution as it would be fully PBI.

 - Can Powerquery take a snapshot of a dataset and store the results within a PBI Calculated table before the API overwrites the latest dataset?

- could i append the table with a snapshot date to control the clearing of old data?

- also schedule the snapshot, example only do the snapshot every 3 months?

Or can powerquery write the table results to temp location and then compare a previous vs latest?

Please Any Help, i have to make this work?

Regards

[lbendlin]

yeah, no.  Power BI has no memory* . You need to manage your snapshots in the upstream system. Keep storing your snapshots as CSV.

* Yes, there is the semantic model self referencing option in Power Query but that has no safety net, and it will break your brain.

[icassiem]

Thank you @lbendlin 

I've been thinking

1. Have a control callendar table with start of month

2. Control table date = current update 

, make table prev snapshot date statis as Prev

3. Using g powerquery condition when to import data from another dataset, current day = control table with snapshot date and set status = curr

Just thinking of logic, but I need to know how to control the clearing of old PREV records etc

Any thoughts of my logic thinking?

[lbendlin]

Still no. Use external storage.

[icassiem]

@lbendlin  oh gosh 🙂 

Cleint does not have storage and i need to find an automated method

The client managed to do the prev vs latest ,ethod by maintaining an csv export of all the data, but now we created an API of latest only

Can PBI automatically store the data externally by creating/created csv and storing snapshots into the file, other than perhaps configuring the csv file, i would like to make the export to csv automated?

or should i rather start of by setting up an excel file that connects to the API and have formulas etc to create the logic of previous but that would also require a manual refresh before importing?

there's no system in between 😞

any ideas please?

[lbendlin · Accepted solution]

the answer is still no. Power BI is not a data storage solution. And it doesn't sound like Fabric is an option for you either.

[icassiem]

Thank You Very Much for Helping @lbendlin 

Not sure how now 😞