Dataflow Incremental Refresh in Premium

[hnguyen76]

Good morning!

I'm beginning to dive into dataflows using a premium environment. If I understood incremental refreshes correctly, only new data should get refreshed, right? I think with my current setup, the entire entity gets refreshes on a scheduled refresh. I think. If not, then my question is: Is there a way to create a column maybe called "CreatedOn" and show when the data was added to the entity?

The setup:
1. Create custom entity for dataflows with initial one month of historical data from a folder on a server.
2. Single table is supplied / consumed with two datetime columns: Date and LastRefreshed:

Date = Table.AddColumn(#"Filtered Rows", "Date", each DateTime.FromText([Month] & "/1/" & [Year]), type datetime),
  LastRefreshed = Table.AddColumn(Date, "LastRefreshed", each DateTime.From(DateTimeZone.ToText(DateTimeZone.FixedUtcNow(), "dd-MMM-yyyy hh:mm tt")), type datetime),
  #"HFMData-44617465-autogenerated_for_incremental_refresh" = Table.SelectRows(LastRefreshed, each DateTime.From([Date]) >= RangeStart and DateTime.From([Date]) < RangeEnd)
in
  #"HFMData-44617465-autogenerated_for_incremental_refresh"

3. Initial Refresh + add incremental refresh settings:

4. Connected to dataflow and check first dataset last refreshed:

5. Added new data to folder and waited for scheduled refresh. From what I can understand, the second refresh shouldn't change the values of the unchanged data but upon second refresh it seems to have:

Looking into the parameters supplied by the auto-generated RangeStart and RangeEnd I see "1/1/1901" and "12/31/9999" so I'm assuming it's picking up all the data within that range on refresh? Please correct me and guide me in the right direction if I'm wrong.

[v-chuncz-msft]

@hnguyen76 

You may try using original column instead of a custom column to detect data changes.

[hnguyen76]

Hi @v-chuncz-msft 
When you say original column instead of custom column. What do you mean?