topic Re: Dataflow Gen2 incremental refresh alternative to modified date in Dataflow

Dataflow Gen2 incremental refresh alternative to modified date

PowerBITestingG — Thu, 14 Aug 2025 08:18:17 GMT

We need hourly updates in a dataflow that writes to warehouse , but the source has no “modified date,” so Gen2 incremental refresh can’t detect changes. Full table refresh every hour is too heavy.

Approach:

Historical dataflow: all older data, refreshed weekly.
Recent dataflow: last 3 months, refreshed hourly.
Create Union view in warehouse
Use filters to avoid duplicates: historical < 90 days ago, recent >= 90 days ago.

Goal: Hourly updates for recent data while keeping historical

Question: Is this a good pattern for updates when no modified date exists?

Re: Dataflow Gen2 incremental refresh alternative to modified date

v-veshwara-msft — Thu, 14 Aug 2025 17:44:44 GMT

Hi PowerBITestingG ,
Thanks for posting in the Microsoft Fabric Community.

Your approach works well when there is no modified date to use for incremental refresh. The filters you mentioned for separating historical and recent data are a good way to avoid duplicates, and the union view in the warehouse can combine them.

If your source can have late-arriving records, you can keep a small overlap in the date ranges and then remove any duplicates in the union view using a unique key.

Thanks for sharing here as this helps others in the community, and please reach out for any further assistance.

Re: Dataflow Gen2 incremental refresh alternative to modified date

v-veshwara-msft — Mon, 18 Aug 2025 09:33:49 GMT

Hi PowerBITestingG ,
Just following up on your query. If further assistance is needed, please reach out.
Thank you.

Re: Dataflow Gen2 incremental refresh alternative to modified date

v-veshwara-msft — Fri, 22 Aug 2025 05:14:49 GMT

Hi PowerBITestingG ,

Following up to see if your query is resolved and if any of the responses helped.
If you're still facing issues, feel free to reach out.

Thank you.

Re: Dataflow Gen2 incremental refresh alternative to modified date

Shahid12523 — Sat, 23 Aug 2025 13:39:52 GMT

Problem: No “modified date” → Gen2 incremental refresh can’t detect changes.

Your approach (historical flow weekly + recent flow hourly + union in warehouse) is a common partitioning strategy when incremental keys are missing.

Key considerations:

Make sure both flows use mutually exclusive filters (<90 days vs ≥90 days) to prevent duplicates.

Test for late-arriving records (data older than 90 days that may still update). If that happens, adjust the cutoff window.

Monitor performance of the hourly flow; keep recent partition as narrow as possible.

Overall: Good pattern until source provides a change-tracking field.

Re: Dataflow Gen2 incremental refresh alternative to modified date

v-veshwara-msft — Tue, 26 Aug 2025 06:44:24 GMT

Hi PowerBITestingG ,
Just wanted to check if the responses provided were helpful. If further assistance is needed, please reach out.
Thank you.