Solved: Dataflow Capture Snapshots

FarhanMalik · ‎08-12-2021

Hi All,

I have a dataflow which is connected to a organization wide database. Now the requirement is to take snapshots of the data as of different dates and times so changes can be compared. A manual solution is that i take csv dump of dataflow results at every refresh and then compare. However, i would like to automate. Any advise, direction, hint or link of any useful resource would be highly apprecoated.

lbendlin · ‎08-13-2021

Change your direction of thought. Keep in mind that dataflows are already csv-like storage blobs. When you take your snapshots of the organization wide database contents you stuff these snapshots into the dataflow. Then do your intended reports in Power BI as needed.

WARNING: There is no guarantee that dataflows will hold your data forever. You may lose all of the history when you have to rebuild the dataflow for whatever reason. Either store your snapshots on-premise, or abandon the idea altogether and move on to the much more efficient CDC (Change Data Capture) philosophy where you keep track of changes in the data rather than wasting space with unreliable (and un-recoverable) snapshots.

View solution in original post

lbendlin · ‎08-13-2021

Change your direction of thought. Keep in mind that dataflows are already csv-like storage blobs. When you take your snapshots of the organization wide database contents you stuff these snapshots into the dataflow. Then do your intended reports in Power BI as needed.

WARNING: There is no guarantee that dataflows will hold your data forever. You may lose all of the history when you have to rebuild the dataflow for whatever reason. Either store your snapshots on-premise, or abandon the idea altogether and move on to the much more efficient CDC (Change Data Capture) philosophy where you keep track of changes in the data rather than wasting space with unreliable (and un-recoverable) snapshots.