Incremental Refresh Issue with UPDATE_DATE_TIME Field — Data Mismatch in Power BI

[manoj_0911]

Hi everyone,

I'm facing a data mismatch issue while implementing incremental refresh in Power BI, and I’d really appreciate your insights.

 

• Our IVR data comes from multiple tables.

• There are two datetime fields in our dataset:

  • TXN_DATE: From DATE_TIME table – 15-minute interval datetime, used for reporting and filters.

  • UPDATE_DATE_TIME: From AG table – timestamp for when data was last updated.

---

📌The Context:

• Historically, we’ve used TXN_DATE in Power BI for filters and reporting — everything worked fine.

• However, there’s a small (~1%) data mismatch with the backend source since some IVR records (especially email or long-call records) get updated days after the actual interaction (delayed processing).

• To fix this, our DB admin recommended using UPDATE_DATE_TIME for incremental refresh, so any late-arriving or updated data would be picked up.

---

⚠️The Problem:

• After switching to UPDATE_DATE_TIME for incremental refresh:

  • Power BI refresh completes successfully (every 2 hours).

  • But data is no longer matching what we expect from the DB or validation source.

• Using TXN_DATE gives better data accuracy on visuals but misses late updates.

• We cannot do a full refresh frequently due to dataset size.

• In Tableau, we solved this using a combined full + incremental refresh. But in Power BI, it seems only incremental is allowed.

---

❓My Questions:

1. Has anyone faced a similar issue where late-arriving data affects incremental refresh?

2. How can I ensure accurate incremental refresh using UPDATE_DATE_TIME but still filter/report using TXN_DATE for user-facing visuals?

3. Is there a best practice to handle this situation in Power BI – perhaps by:

   • Using UPDATE_DATE_TIME only in RangeStart / RangeEnd filters

   • But continuing to use TXN_DATE for slicers and reporting?

4. Any workaround to do a partial full + partial incremental refresh like in Tableau?

---

🛠️ What I’ve Tried:

• Used UPDATE_DATE_TIME for incremental refresh partitioning.

• Refresh works, but report visuals don’t match expectations.

• Reverting back to TXN_DATE gives accurate visuals but doesn’t pick up updated data timely.

---

Would really appreciate your experience or suggestions on how to handle this.

Thanks in advance!
— Manoj Prabhakar

[v-lgarikapat · Accepted solution]

Hi @manoj_0911 ,

Thanks for reaching out to the Microsoft fabric community forum

@rajendraongole1 ,

Thanks for your prompt response

In addition to @rajendraongole1 , below are a few workarounds

Enable Detect Data Changes Option

 You’ve already done this good move.

This ensures Power BI checks for changes in the UPDATE_DATE_TIME column even within existing partitions.

 Add a Buffer to the Refresh Window

In your image, you’ve set the incremental refresh to start 2 days before the current date. That’s smart but consider extending it to 3–5 days if updates are delayed longer.

This helps catch updates that arrive slightly late but still within the buffer.

Enhancements

Create a Composite Key for Change Detection

If your table has a unique ID (TXN_ID), consider creating a hash or composite key

Text.Combine({Text.From([TXN_ID]), Text.From([UPDATE_DATE_TIME])}, "|")

Use this for change detection to ensure Power BI identifies updated rows.

Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn

We truly appreciate your continued engagement and thank you for being an active and valued member of the community.

If you're still experiencing any challenges, please don’t hesitate to reach out  we’d be more than happy to assist you further.

We look forward to hearing from you.

Best regards,
Lakshmi

[v-lgarikapat]

Hi @manoj_0911 ,

I wanted to follow up and confirm whether you’ve had the opportunity to review the information we provided. If you have any questions or need further clarification, please don’t hesitate to reach out.

We truly appreciate your continued engagement and thank you for being an active and valued member of the community.

Best Regards,

Lakshmi

[v-lgarikapat]

Hi @manoj_0911 ,

I just wanted to check if your issue has been resolved. If you still have any questions or need help, feel free to reach out I’m happy to assist.

Thank you for being an active part of the community. Looking forward to hearing from you!

Best regards,
Lakshmi

[v-lgarikapat]

Hi @manoj_0911 ,

I just wanted to check if your issue has been resolved. If you still have any questions or need help, feel free to reach out I’m happy to assist.

Thank you for being an active part of the community. Looking forward to hearing from you!

Best regards,
Lakshmi

[rajendraongole1]

Hi @manoj_0911  - you should keep TXN_date in the dataset for reporting, date slicers, visual filtering, etc.dont use this TXN_DATE not be used for incremental filtering. you can try and use Use UPDATE_DATE_TIME to drive incremental refresh.

One more, pbi does not natively support the Tableau-style "partial full" + incremental blend in one refresh cycle. Please check and let know.