Forum Discussion
Dataflow Gen2 refresh issue - Dynamics 365 entity to warehouse table
Hi All, I have created a Dataflow Gen2 which is picking up data from the 'account' entity in Dynamics 365. The M-Code for this query is as follows: let
Source = CommonDataService...
In this business, new accounts are added on almost daily basis - thus, I would only like to ingest new and updated accounts each day, rather than the entire 2.3 million+ each day
lbendlin
Super User
Super User
thus, I would only like to ingest new and updated accounts each day
Thank you for the background information, that helps.
thus, I would only like to ingest new and updated accounts each day
Oh, you're doomed. It's called Incremental refresh for a reason. Data is expected to be immutable. If it isn't (like your "updated accounts" ) then Incremental refresh will not work for you, and you will end up with multiple versions of the same account across multiple partitions. Instead you would need to implement your own differential refresh process, CDC or SCD2.