Forum Discussion

petersmith's avatar
petersmith
New Member
7 years ago

Dataflow incremental refresh with primary key

Hi All,   Loving dataflows, but would love the ability to do an incremental refresh in more of an 'upsert' style. That is, maintain the uniqueness of a primary key within the dataset. Any ideas on ...
Dataflow
Need Help
Anonymous's avatar
Anonymous
Not applicable
7 years ago

I SO GLAD someone asked this question too!

Currently incremental refresh works for the data was updated based on datetime stamp, say last 10 days. It does not take into consideration the Primary Key that might have been inserted in the past.
Eg: lets say a "Sales Order" was created 3 months with a Status "Open", but only today its status was changed to "Shipped". Since I am only looking for past 10 days data for incremental refresh, a new Sales Order row gets inserted into the Dataset causing my "Sales Order" table to be duplicated.

How can we avoid this? Ideally along with the Datetime stamp, there needs to be a option where we specify the Unique Key column too. If incremental refresh contains any of the Unique Keys loaded in the past, they need to be deleted too and reinserted.

  • Anonymous's avatar
    Anonymous
    Not applicable
    6 years ago

    Hey SamRock,

     

    Any update regarding consideration of primary key when incrementally refreshing? I have a similar situation and I am not sure Power BI dataflows can solve my issue.

    • GilbertQ's avatar
      GilbertQ
      Super User
      6 years ago
      Hi there

      Currently incremental refresh only works with dates and unfortunately not with a primary key
      • Anonymous's avatar
        Anonymous
        Not applicable
        6 years ago

        Is there any timeline for if/when it will be added as a feature for dataflows? This can be seen as a critical feature for refreshing data dependent on past state changes (i.e. Orders, Status, Deliveries, Inventory, Requests, etc)