Forum Discussion
Missing some data for incremental refresh after schedule refresh
Hi suyoggunjal ,
Thank you for reaching out to the Microsoft Fabric Community Forum. Also thank you rohit1991 and pankajnamekar25 for your inputs.
Based on your issue, it seems that the missing Shift 3 data after the scheduled refresh is likely due to timestamp boundaries. When Shift 3 spans across midnight, the RangeStart and RangeEnd filters used for incremental refresh may not fully capture this data.
Here are the recommended steps to follow which helps in resolving the issue:
- Ensure timestamps for Shift 3 align correctly with your incremental refresh period, especially if the shift extends past midnight.
- Review the partitioning logic to ensure it includes the full date range for Shift 3, especially across days.
- Trigger a full refresh to confirm if the issue is tied to the incremental refresh setup.
- Consider expanding your RangeStart/RangeEnd parameters to capture late-night data properly.
I hope this will resolve your issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi suyoggunjal ,
Thank you for reaching out to the Microsoft Fabric Community Forum. Also thank you rohit1991 and pankajnamekar25 for your inputs.
Based on your issue, it seems that the missing Shift 3 data after the scheduled refresh is likely due to timestamp boundaries. When Shift 3 spans across midnight, the RangeStart and RangeEnd filters used for incremental refresh may not fully capture this data.
Here are the recommended steps to follow which helps in resolving the issue:
- Ensure timestamps for Shift 3 align correctly with your incremental refresh period, especially if the shift extends past midnight.
- Review the partitioning logic to ensure it includes the full date range for Shift 3, especially across days.
- Trigger a full refresh to confirm if the issue is tied to the incremental refresh setup.
- Consider expanding your RangeStart/RangeEnd parameters to capture late-night data properly.
I hope this will resolve your issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi suyoggunjal ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.