Incremental Refresh Overlapping/Duplication of Rows

[Arun_Benny_TS]

Dear Team,

I am facing an issue with the incremental refresh, when I am opting for incremental refresh starting from one month before the refresh date, the data of the first day of the month is twice the actual value. 

Example : 

When I opt for this setting, the data of first of March is twice the actual value, no matter the how many refresh has been done. And all the data afterwards that particular date is correct though. 

Below is the screenshot of the parameter settings and the m-query.

Please do support with a possible solution for the issue. 

[v-kpoloju-msft]

Hi @Arun_Benny_TS,

Thank you for reaching out to the Microsoft fabric community forum. Thank you @mariussve1, for your inputs on this issue.

After thoroughly reviewing the details you provided, I have identified few workarounds that may help resolve the issue. Please follow these steps:

The Incremental Refresh Overlapping/Duplication of Rows error in Power BI usually occurs due to improper partitioning or misaligned filter logic during the implementation of Incremental Refresh.

This problem often arises when the RangeStart and RangeEnd parameters are not correctly aligned with the data source's partitions, leading to overlapping data segments or duplicated rows. Ensuring precise filter alignment and verifying partition boundaries can help mitigate this issue.

Ensure your query correctly applies the RangeStart and RangeEnd parameters in a date-based filter. Check Applied Steps in Power Query by navigating to Transform Data → Check Filtered Rows.

Ensure that the date filter applied in Power Query (RangeStart and RangeEnd parameters) correctly partitions the data. Typically, it should be: FilteredRows = Table.SelectRows(Source, each [Date] >= RangeStart and [Date] < RangeEnd). The less than (<) RangeEnd is crucial to avoid overlapping.

If using a native SQL query, ensure the filter is correctly applied in the WHERE clause: “WHERE DateColumn >= @RangeStart AND DateColumn < @RangeEnd.” Avoid <= RangeEnd, as it may cause overlapping rows.

Ensure the RangeStart and RangeEnd parameters are correctly set. The RangeStart should filter dates AFTER the specified date, and the RangeEnd should filter dates BEFORE OR EQUAL TO the specified date. This prevents overlapping intervals that can cause duplication.

The document below indicates that the refresh fails due to duplicate values,

Kindly review the documentation link provided for a better understanding:

Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn

I hope this could resolve your issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you for using Microsoft Community Forum.

[v-kpoloju-msft]

Hi @Arun_Benny_TS,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-kpoloju-msft]

Hi @Arun_Benny_TS,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-kpoloju-msft]

Hi @Arun_Benny_TS,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[mariussve1]

Hi,

The filter on RangeStart and RangeEnd needs to be:

>= RangeStart

< RangeEnd

Looks like you are using >= and <=
Please change to <