Missing some data for incremental refresh after schedule refresh

[suyoggunjal]

I'm noticing that data for Shift 3 is missing for the current month after setting up the incremental refresh, even though shift-wise data is available in the dataset

[v-tsaipranay]

Hi @suyoggunjal ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-tsaipranay]

Hi @suyoggunjal ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-tsaipranay · Accepted solution]

Hi @suyoggunjal ,

Thank you for reaching out to the Microsoft Fabric Community Forum. Also thank you @rohit1991  and @pankajnamekar25  for your inputs.

Based on your issue, it seems that the missing Shift 3 data after the scheduled refresh is likely due to timestamp boundaries. When Shift 3 spans across midnight, the RangeStart and RangeEnd filters used for incremental refresh may not fully capture this data.

Here are the recommended steps to follow which helps in resolving the issue:

• Ensure timestamps for Shift 3 align correctly with your incremental refresh period, especially if the shift extends past midnight.
• Review the partitioning logic to ensure it includes the full date range for Shift 3, especially across days.
• Trigger a full refresh to confirm if the issue is tied to the incremental refresh setup.
• Consider expanding your RangeStart/RangeEnd parameters to capture late-night data properly.

I hope this will resolve your issue, if you need any further assistance, feel free to reach out.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.

[v-tsaipranay]

Hi @suyoggunjal ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[rohit1991]

Hi @suyoggunjal ,

If you're missing data for Shift 3 after configuring incremental refresh in Power BI, it's likely due to how the incremental refresh policy is set up, particularly the parameters for the range of data being refreshed. Incremental refresh only processes data within a defined date range, typically using a date column like Date or ModifiedDate.

If Shift 3 data falls outside this range—for example, if it's logged late at night and timestamped past midnight—it might not be included in the current refresh window. Ensure that your partitioning column used for incremental refresh includes all shifts properly, and that the data for Shift 3 is timestamped correctly and falls within the defined refresh period. Also, verify that your dataset contains all expected records before the refresh, and consider performing a full refresh to test if the issue is tied to the incremental policy setup.

[suyoggunjal]

Hi @rohit1991 ,

Thanks for the input.

Before publishing, I'm able to see the data for the same date range—including Shift 3. However, after publishing and triggering the scheduled refresh, the data for the current month (specifically Shift 3) goes missing. For context, I'm archiving data for the last 3 months and set the incremental refresh to refresh only the last 1 day of data.

[pankajnamekar25]

Hello @suyoggunjal 

Double-check that the filter used in Power Query (usually RangeStart and RangeEnd) includes all shifts, especially late-night entries crossing into the next day.

Thanks,
 Pankaj Namekar | LinkedIn

If this solution helps, please accept it and give a kudos (Like), it would be greatly appreciated.

[suyoggunjal]

Hi @pankajnamekar25 ,

Thanks for the input.

Before publishing, I'm able to see the data for the same date range—including Shift 3. However, after publishing and triggering the scheduled refresh, the data for the current month (specifically Shift 3) goes missing. For context, I'm archiving data for the last 3 months and set the incremental refresh to refresh only the last 1 day of data.