Forum Discussion
Missing some data for incremental refresh after schedule refresh
I'm noticing that data for Shift 3 is missing for the current month after setting up the incremental refresh, even though shift-wise data is available in the dataset
Hi suyoggunjal ,
Thank you for reaching out to the Microsoft Fabric Community Forum. Also thank you rohit1991 and pankajnamekar25 for your inputs.
Based on your issue, it seems that the missing Shift 3 data after the scheduled refresh is likely due to timestamp boundaries. When Shift 3 spans across midnight, the RangeStart and RangeEnd filters used for incremental refresh may not fully capture this data.
Here are the recommended steps to follow which helps in resolving the issue:
- Ensure timestamps for Shift 3 align correctly with your incremental refresh period, especially if the shift extends past midnight.
- Review the partitioning logic to ensure it includes the full date range for Shift 3, especially across days.
- Trigger a full refresh to confirm if the issue is tied to the incremental refresh setup.
- Consider expanding your RangeStart/RangeEnd parameters to capture late-night data properly.
I hope this will resolve your issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
8 Replies
Hi suyoggunjal ,
Thank you for reaching out to the Microsoft Fabric Community Forum. Also thank you rohit1991 and pankajnamekar25 for your inputs.
Based on your issue, it seems that the missing Shift 3 data after the scheduled refresh is likely due to timestamp boundaries. When Shift 3 spans across midnight, the RangeStart and RangeEnd filters used for incremental refresh may not fully capture this data.
Here are the recommended steps to follow which helps in resolving the issue:
- Ensure timestamps for Shift 3 align correctly with your incremental refresh period, especially if the shift extends past midnight.
- Review the partitioning logic to ensure it includes the full date range for Shift 3, especially across days.
- Trigger a full refresh to confirm if the issue is tied to the incremental refresh setup.
- Consider expanding your RangeStart/RangeEnd parameters to capture late-night data properly.
I hope this will resolve your issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi suyoggunjal ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.
Hello suyoggunjal
Double-check that the filter used in Power Query (usually RangeStart and RangeEnd) includes all shifts, especially late-night entries crossing into the next day.
Thanks,
Pankaj Namekar | LinkedInIf this solution helps, please accept it and give a kudos (Like), it would be greatly appreciated.
Hi pankajnamekar25 ,
Thanks for the input.
Before publishing, I'm able to see the data for the same date range—including Shift 3. However, after publishing and triggering the scheduled refresh, the data for the current month (specifically Shift 3) goes missing. For context, I'm archiving data for the last 3 months and set the incremental refresh to refresh only the last 1 day of data.
Hi suyoggunjal ,
If Shift 3 spans midnight, the incremental refresh partitioning (which works off a datetime column like Date or ModifiedDate) can end up excluding those records if the partition boundaries don’t line up perfectly with your shift schedule. For example, if your RangeStart and RangeEnd are set to filter only by date, anything logged after midnight might not get picked up as expected for “Shift 3”.
-
Check the datetime column used for incremental refresh make sure it truly covers the whole Shift 3 window, including times after midnight.
-
Query folding matters: In Power Query, right-click your last step and make sure “View Native Query” is available. If it’s not, folding has broken and Power BI can’t push filters to the source efficiently, which can mess with incremental refresh.
-
Full refresh vs incremental refresh: Run a full refresh and compare it with your incremental. If full picks up all your shifts and incremental does not, it’s almost always a partition boundary issue.
-
Time zones: Double-check if your data is stored in UTC or local time sometimes data just past midnight appears to “disappear” due to time zone offsets.
Focus on how your incremental partitions line up with your shift logic, make sure query folding isn’t broken, and audit for time zone mismatches.Hi rohit1991 ,
Thanks for the input.
Before publishing, I'm able to see the data for the same date range—including Shift 3. However, after publishing and triggering the scheduled refresh, the data for the current month (specifically Shift 3) goes missing. For context, I'm archiving data for the last 3 months and set the incremental refresh to refresh only the last 1 day of data.
-
Hi suyoggunjal ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.
Hi @suyoggunjal ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.
