Forum Discussion

pws_flick's avatar
pws_flick
Frequent Visitor
1 year ago

Dataflow Gen2 refresh issue - Dynamics 365 entity to warehouse table

Hi All,  I have created a Dataflow Gen2 which is picking up data from the 'account' entity in Dynamics 365.    The M-Code for this query is as follows:     let Source = CommonDataService...
Dataflow
Need Help
lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
1 year ago

Out of curiosity - why are you enabling incremental refresh on a dimension table?

pws_flick's avatar
pws_flick
Frequent Visitor
1 year ago

In this business, new accounts are added on almost daily basis - thus, I would only like to ingest new and updated accounts each day, rather than the entire 2.3 million+ each day 

  • lbendlin's avatar
    lbendlin
    Icon for Super User rankSuper User
    1 year ago

     

     thus, I would only like to ingest new and updated accounts each day

     

    Thank you for the background information, that helps.

     

     

     thus, I would only like to ingest new and updated accounts each day

     

    Oh, you're doomed.  It's called Incremental refresh for a reason. Data is expected to be immutable. If it isn't (like your "updated accounts" ) then Incremental refresh will not work for you, and you will end up with multiple versions of the same account across multiple partitions.   Instead you would need to implement your own differential refresh process, CDC or SCD2.