Forum Discussion

petersmith's avatar
petersmith
New Member
7 years ago

Dataflow incremental refresh with primary key

Hi All,   Loving dataflows, but would love the ability to do an incremental refresh in more of an 'upsert' style. That is, maintain the uniqueness of a primary key within the dataset. Any ideas on ...
Dataflow
Need Help
Anonymous's avatar
Anonymous
Not applicable
6 years ago

Hey SamRock,

 

Any update regarding consideration of primary key when incrementally refreshing? I have a similar situation and I am not sure Power BI dataflows can solve my issue.

GilbertQ's avatar
GilbertQ
Super User
6 years ago
Hi there

Currently incremental refresh only works with dates and unfortunately not with a primary key
  • Anonymous's avatar
    Anonymous
    Not applicable
    6 years ago

    Is there any timeline for if/when it will be added as a feature for dataflows? This can be seen as a critical feature for refreshing data dependent on past state changes (i.e. Orders, Status, Deliveries, Inventory, Requests, etc)

    • markive's avatar
      markive
      Advocate II
      6 years ago

      The underlying technology for incremental refresh seems to be based on Partitions from SSAS. Therefore my guess is I don't think this will fundamentally change to support a primary key upsert any time soon.

       

      Assuming therefore that when a record is updated it will appear in your incremental refresh therefore creating a duplicate you should add extra steps in your MCode to remove duplicates keeping the Max Primary Key therefore pruning the data in your Dataflows ETL process.

      • GilbertQ's avatar
        GilbertQ
        Super User
        6 years ago
        Hi there

        I do not think that this is something that will be happening anytime soon.

        You could have an additional column which is the modified datetime which could be used to update?