Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago

Incremental refresh and Group By in M/PowerQuery

Hey all,

 

just started to try out the incremental refresh, but I have a question:

 

Imagine I have a data source that contains all transactions per store per day. I set up the datetime filter that incremental refresh requires. After that I use group by in powerquery to display the data only per store, leaving out the date column.

 

I then publish this report to service, and set a refresh schedule.

 

My question is: what happens in the service? Does incremental refresh still make sense in this instance? It seems to me that because you did a group by, and powerBI only stores the end result of the query, and not the intermediate steps, it would still need to do a full refresh of all your data every time, and the incremental refresh would not work at all.

 

Is that assumption correct?

 

Thanks!

Jaap

Need Help

4 Replies

    • Anonymous's avatar
      Anonymous
      Not applicable
      6 years ago

      So I want to remove the date fields because I want to aggregate users by Country. If I leave in the date field, it will do unique userID counts per day, but that means a returning user would be counted twice in the final table. By removing the date field and aggregating to Country, the unique counts are correct.

       

      I could also just load the entire source table, but depending on the client, data (not in this table, but in others) does run into the billions of rows territory.

       

      And yes, so I understand that incremental refresh is based on a date field. I just think that if the user removes the date field in powerquery, maybe the incremental refresh switch (when you right click on a table) should be greyed out instead of active. It also still mentions the date column that I have removed, which seems to indicate that this would still work, see screenshot:

       

      • Anonymous's avatar
        Anonymous
        Not applicable
        6 years ago

        Hi Anonymous,

        Maybe you can try to create a new query to refer from the current query table and remove date field and grouping in new table, then cancel the loading of original query table. After these, the date fields should exist in the original table that not displayed.
        Since the official document also not explain this cleanly, you can consider contacting to power bi team to know more about incremental refresh.

        Regards,

        Xiaxoin Sheng