Incremental Refresh is Creating Duplicates for specific Timestamps

[AlexNTH]

I believe I've found a bug in incremental refresh because I haven't been able to find a clear explaination for a duplicate value generation. Despite not having an explaination, the bug is consistently reproducable.

I have a table called LINECHART, and the table rows should be unique on the pairing of the columns TIME_TM (which is a datetime column with 10 minute timeslices) and MACHINE (machine name that we're doing part counts for. I work for automated assembly company. For this particular data set there are 26 machines)

The incremental refresh is active for LINECHART and is set to archieve 3 years, and update the last 3 days, and detect changes in TIME_TM, the only time column in that table.

When running the refresh from desktop, there are no duplications (if I'm not wrong, refreshing on desktop does not trigger incremental refresh)

When I publish it to a workspace and run the refresh, there are row duplications for this table for the following conditions:
00:00AM/12:00AM UTC Time value from TIME_TM 
AND
It is the part of JULY 2022

For some reason this duplication only occurs for JULY 2022, or perhaps whatever the current MONTH,YEAR combination it currently is (you can check the date of this post was in July 2022). The rest of the ~2 years of data is perfectly fine.

I've attempted adding a 'table.distinct()' step in the PowerQuery M transformation for LINECHART, which I assumed runs after an incremental refresh, but this had no effect.

I've tried removing incremental refresh, and this resolves the issue. However, this isn't a viable option for LINECHART, as it is a FACT table (referring to STAR SCHEMA) with 2 million rows and must be incrementally refresh to not use up the Premium Capacity's compute load.

It should be noted that the issue shouldn't be caused by the source data being updated, because the issue is happened to rows that are outside of the 3day incremental refresh range.

I've been trying to find the root cause of this issues for a couple days now with no clear culprit. I'm open to suggestions or other troubleshooting techniques.

I thought I'd also include the relationship model. It should be noted that the relationship between Datepicker and TimeZoneMapping is actually one-to-one, because there is always a filter to select exactly one TimezoneOffset. It should also be noted the issue was occuring prior to the table even being imported in, so it shouldn't have an effect on the issue.

[v-cazheng-msft]

Hi @AlexNTH,

For this issue, I would suggest you creating a support ticket. Then, there will be an engineer to have a remote session with you to troubleshoot on this issue.

The link of Power BI Support: Support | Microsoft Power BI

For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Power BI Community

Best Regards,

Community Support Team _ Caiyun

[attonrasquinha]

Hi @AlexNTH ,

Did you find a solution for this issue? Even am facing the same problem - Refresh doesnt create duplicates in Power bi desktop but creates duplicates for specific timestamps after initial refresh in power bi service. Any help will be highly appreciated. Thank you

I have configured incremental refresh using stored procedure. Removing the IR - Doesnt create duplicates in Power BI service.