Dataflow Gen2 incremental refresh alternative to modified date

[PowerBITestingG]

We need hourly updates in a dataflow that writes to warehouse , but the source has no “modified date,” so Gen2 incremental refresh can’t detect changes. Full table refresh every hour is too heavy.

Approach:

• Historical dataflow: all older data, refreshed weekly.

• Recent dataflow: last 3 months, refreshed hourly.

• Create Union view in warehouse

• Use filters to avoid duplicates: historical < 90 days ago, recent >= 90 days ago.

Goal: Hourly updates for recent data while keeping historical

Question: Is this a good pattern for updates when no modified date exists?

[v-veshwara-msft · Accepted solution]

Hi @PowerBITestingG ,
Thanks for posting in the Microsoft Fabric Community.

Your approach works well when there is no modified date to use for incremental refresh. The filters you mentioned for separating historical and recent data are a good way to avoid duplicates, and the union view in the warehouse can combine them.

If your source can have late-arriving records, you can keep a small overlap in the date ranges and then remove any duplicates in the union view using a unique key.

Thanks for sharing here as this helps others in the community, and please reach out for any further assistance.

[v-veshwara-msft]

Hi @PowerBITestingG ,
Just wanted to check if the responses provided were helpful. If further assistance is needed, please reach out.
Thank you.

[Shahid12523 · Accepted solution]

Problem: No “modified date” → Gen2 incremental refresh can’t detect changes.

Your approach (historical flow weekly + recent flow hourly + union in warehouse) is a common partitioning strategy when incremental keys are missing.

Key considerations:

Make sure both flows use mutually exclusive filters (<90 days vs ≥90 days) to prevent duplicates.

Test for late-arriving records (data older than 90 days that may still update). If that happens, adjust the cutoff window.

Monitor performance of the hourly flow; keep recent partition as narrow as possible.

Overall: Good pattern until source provides a change-tracking field.

[v-veshwara-msft]

Hi @PowerBITestingG ,

Following up to see if your query is resolved and if any of the responses helped.
If you're still facing issues, feel free to reach out.

Thank you.

[v-veshwara-msft]

Hi @PowerBITestingG ,
Just following up on your query. If further assistance is needed, please reach out.
Thank you.

[v-veshwara-msft · Accepted solution]

Hi @PowerBITestingG ,
Thanks for posting in the Microsoft Fabric Community.

Your approach works well when there is no modified date to use for incremental refresh. The filters you mentioned for separating historical and recent data are a good way to avoid duplicates, and the union view in the warehouse can combine them.

If your source can have late-arriving records, you can keep a small overlap in the date ranges and then remove any duplicates in the union view using a unique key.

Thanks for sharing here as this helps others in the community, and please reach out for any further assistance.