Forum Discussion
Incrementally load new data without processing all dataset
HI mribeiro ,
As I am sure you know, the incremental refresh is using a date column to determine the "incremental" data to refresh and it is looking for changes in data since that date. Based on that date parameter the system will not go back in time to find previous sales transactions for that specific client since that would be "archived data" already and not picked up.
If I understand your description this is exactly what is happening.
I don't think that there is a particularly easy programmatic way to get around this issue. If this is something that happens infrequently the way I would do this is to create a new query - similar to yours - where you go and JUST get the new client data, then, I would append this to the master sales table and that way you have the historical data available to you and going forward the new client is picked up with the incremental refresh.
Let me know your thoughts.
Hello collinq ,
Thanks for your reply!
Exactly.. you got the issue.
You had a great idea. I followed your suggestion, but unffortunelly I came cross other issue: It's not possible append data in a already incremental configurated table. It's simillar of this issue: https://community.powerbi.com/t5/Service/Incremental-refresh-from-Azure-DataLake/m-p/559711#M56914
You have some suggestion to get around this new issue?
Thanks.
HI mribeiro ,
Hmmmmmm, ok, perhaps you could do part (or all) of this in a DataFlow? You could have the "old" stuff in the dataflow and not necessarily refresh it every day? OR, you could do the new stuff in the dataflow? Either way, if half the work is done in a dataflow and then you just combine the data together in the final report I think that should work.