Dataflow Gen2 incremental refresh alternative to modified date

[PowerBITestingG]

We need hourly updates in a dataflow that writes to warehouse , but the source has no “modified date,” so Gen2 incremental refresh can’t detect changes. Full table refresh every hour is too heavy.

Approach:

• Historical dataflow: all older data, refreshed weekly.

• Recent dataflow: last 3 months, refreshed hourly.

• Create Union view in warehouse

• Use filters to avoid duplicates: historical < 90 days ago, recent >= 90 days ago.

Goal: Hourly updates for recent data while keeping historical

Question: Is this a good pattern for updates when no modified date exists?

[v-veshwara-msft]

Hi @PowerBITestingG ,
Just following up on your query. If further assistance is needed, please reach out.
Thank you.

[v-veshwara-msft]

Hi @PowerBITestingG ,
Thanks for posting in the Microsoft Fabric Community.

Your approach works well when there is no modified date to use for incremental refresh. The filters you mentioned for separating historical and recent data are a good way to avoid duplicates, and the union view in the warehouse can combine them.

If your source can have late-arriving records, you can keep a small overlap in the date ranges and then remove any duplicates in the union view using a unique key.

Thanks for sharing here as this helps others in the community, and please reach out for any further assistance.