Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
4 years ago

Adding data from Table2 to Table1 upon refresh

Hello,

 

I'm working on re-building a report that pulls data from ADO via an ODATA feed every night.  the goal of this is to take snapshots of the states of various features every day so that we can use the data for historical purposes (AKA see what got done on time, vs. what just got done).  The person who set up the system before me used the WorkItemsSnapshot feed from ADO, which we're encountering throttling issues on, and the documentation says we shouldn't be using in this way.

 

My approach I want to take is to create a table that pulls in the data from the ODATA feed every night at a scheduled refresh, and then appends that data into a table that has all the previous historical data in it. Is there a simple way to do this?

 

To put it another way; Table 2's contents change every night, and I would like to add a copy of Table 2's contents to Table 1 every night, that way I will have a historical record of when certain changes in the data happened.

 

I'm not against using Python to do this, as I've got skills in that area.  I've looked into incrimental refresh, but from what I've seen it doesn't look like I can use that to be able to look back at historical data to compare easily, but I could be wrong.  Lastly, I'm hoping whatever solution can be found is something that will work with automated refresh from the web version.

3 Replies

  • "Is there a simple way to do this?"

     

    No. There is also no complicated way of doing this in Power BI. Power BI has no memory, and no concept of snapshots. You need to do that further upstream.

    • Anonymous's avatar
      Anonymous
      Not applicable

      I get tht PowerBI has no memory, but is there a way that I can use a function, or Python so that every time a table is loaded, the contents of that loaded table get added to another table?

       

      In the mean time, I'll start exploring ways to capture these snapshots further upstream.

      • lbendlin's avatar
        lbendlin
        Super User

        Appending tables in Power Query is as simple as

         

        Table1 & Table2

         

        (assuming they have the same structure - If not then you need to use Table.Combine or more elaborate functions)