Forum Discussion
Helper IIIAppending Two Tables - One With Incremental Refresh Policy, and One With Full Load
Hi Dave1mo1 ,
It might be becuase the incremental setting is not transferred to the appended table. So, in Power query, it takes that filtered data.ie., the between RangeStart and RangeEnd dates.
In this case, just try the below steps that may work.
1. Take a reference from your second table that requires full refresh
2. then append your first table (incremental refresh table) with the referenced table taken in #1
3. then apply the incremental refresh on that appended table.
Thanks.
both are doing Append but Append as new will take a separate copy and append the data.
Solution SupplierHi Dave1mo1 ,
It might be becuase the incremental setting is not transferred to the appended table. So, in Power query, it takes that filtered data.ie., the between RangeStart and RangeEnd dates.
In this case, just try the below steps that may work.
1. Take a reference from your second table that requires full refresh
2. then append your first table (incremental refresh table) with the referenced table taken in #1
3. then apply the incremental refresh on that appended table.
Thanks.
Helper IIIThis is working to get the table with the incremental load data correctly into the model (thank you!), but it looks like it's duplicating the full-load table across multiple partitions. Any thoughts on that? Otherwise I might just split the "incremental load" table into a "Historical Table," load it once, and disable the data refresh, then load the same table a second time starting from the end date of the "Historical Table" and load that table fully each time. This would essentially remove the need to have the incremental refresh policy, but it's not a long-term solution once the second query becomes so large that it slows down the refresh.
Not sure how else to handle this. Someone on another forum suggested doing a union in DAX to create a calculated table, but that would bloat the size of the model by loading the tables twice (once initially, then a second time when they're unioned), and I would need to have the exact same fields in both tables. Most fields are the same, but there are a few fields that are unique to one table or the other.
Any additional thoughts? Really appreciate your help, by the way.