Forum Discussion

captainlaw's avatar
captainlaw
Microsoft Employee
9 years ago

Refresh Data - Append New Data Only

Hello, I'm wondering is it possible to setup refresh of a data source based on date column, meaning only append new data from last load based on date column.  It would be a good feature to add for ...
General Comment
captainlaw's avatar
captainlaw
Microsoft Employee
9 years ago

In the workaround, assuming here's what we've loaded in the report -

Current dataset 11/1-11/30

Previous dataset 10/1-10/31

 

From the video, so the step is to copy Current and overwrite Previous.

Previous dataset 11/1-11/30

 

Then, we load latest dataset -

Current dataset 12/1-12/15

 

So base on the script, it will refresh Previous + Current = 11/1-12/15

Am I correct in the above assumption?

If so, what happened to dataset 10/1-10/31?  I'm looking for a solution that will keep all data, but only refresh/append new data.

For my scenario to work, can we still only have 2 datasets or do we need to continue to add more?

  • MarcelBeug's avatar
    MarcelBeug
    Community Champion
    9 years ago

    Please note that the workaround will not be suitable for large data sets.

     

    Your assumptions are not correct as new data will be appended to PreviousData each time in Dataset, so Dataset will have all data after refresh.

     

    In your examples:

    In CSV 11/1 - 11/30.

    In PreviousData 10/1 - 10/31.

     

    After refresh, Dataset will have data 10/1 - 11/30.

    Copy this to PreviousData.

     

    New CSV file 12/1 - 12/15.

    After refresh, Dataset will have data 10/1 - 12/15.

    Copy this to PreviousData.

     

    So you'll only have 2 datasets.

    • Pankaj1208's avatar
      Pankaj1208
      Regular Visitor
      8 years ago

      Hi.... Its good but when i replace the value second time then its not update in previous table so please give the solution 

      • powerbiuser101's avatar
        powerbiuser101
        Advocate I
        8 years ago

        This doesn't work for me either. My dataset does not update on the 2nd refresh. This is rather confusing as I had thought the idea was to append data to "Previousdata" from an ever changing "Dataset"? Based on MarcelBeug's reply, sounds like this was the opposite. I don't understand the "Copy this to PreviousData" step. Shouldn't the Merged/Combined step be added to "Previousdata" rather than "Dataset"?

         

         

         

         

        Merged = Table.NestedJoin(PreviousData,{"Date"},NewData,{"Date"},"NewColumn",JoinKind.RightAnti),
    Combined = if Table.IsEmpty(Merged) then PreviousData else Table.Combine({PreviousData, Merged[NewColumn]{0}})
in
    Combined

        In my testing, Combined was also referencing an older Table in Merged[NewColumn]. Which made sense to me why it worked the first time and not any subsequent refresh. Re-adding 2 new additional steps as above worked but this would cause some step-bloat if this has to be done every single time I need to append new data.