Forum Discussion
Incremental Refresh – Archived Partition Overwritten After Refresh
- 4 months ago
Hi JasonKang
Thank you for reaching out to the Microsoft Fabric Forum Community.
Juan-Power-bi Thanks for the inputs.
As per my knowledge, the issue isn’t publishing anymore, it’s how semantic_model_date is being used.Since you’re reassigning this date to different batches every day, the same partition ends up holding different data over time.
Incremental refresh expects that date column to stay consistent so it can manage partitions properly. In your case, because the value keeps changing for different data, Power BI ends up reprocessing that partition during refresh. When that happens, the partition gets rebuilt with the new data, which replaces what was already there instead of keeping the archived data.
Also, there’s no row count or data size limit in Fabric that would cause silent failures, if a partition returns 0 rows, it’s still treated as a successful refresh.
On top of that, since Parquet/Blob sources don’t support query folding, Power BI evaluates data based on the current files. If the source only contains the latest batch, partitions being refreshed may return no data and get overwritten.
So the behavior you’re seeing is actually expected. The refresh is working as designed, but the partition logic breaks because the date being used isn’t stable. Ideally, the column used for RangeStart and RangeEnd should be something fixed (like the actual record date), not something that changes per batch.
If there are any deviations from your expectation please let us know we are happy to address.
Thanks.
Hi JasonKang
Thank you for reaching out to the Microsoft Fabric Forum Community.
Juan-Power-bi Thanks for the inputs.
As per my knowledge, the issue isn’t publishing anymore, it’s how semantic_model_date is being used.
Since you’re reassigning this date to different batches every day, the same partition ends up holding different data over time.
Incremental refresh expects that date column to stay consistent so it can manage partitions properly. In your case, because the value keeps changing for different data, Power BI ends up reprocessing that partition during refresh. When that happens, the partition gets rebuilt with the new data, which replaces what was already there instead of keeping the archived data.
Also, there’s no row count or data size limit in Fabric that would cause silent failures, if a partition returns 0 rows, it’s still treated as a successful refresh.
On top of that, since Parquet/Blob sources don’t support query folding, Power BI evaluates data based on the current files. If the source only contains the latest batch, partitions being refreshed may return no data and get overwritten.
So the behavior you’re seeing is actually expected. The refresh is working as designed, but the partition logic breaks because the date being used isn’t stable. Ideally, the column used for RangeStart and RangeEnd should be something fixed (like the actual record date), not something that changes per batch.
If there are any deviations from your expectation please let us know we are happy to address.
Thanks.
- v-priyankata4 months agoCommunity Support
Hi JasonKang
Thank you for reaching out to the Microsoft Fabric Forum Community.
I hope the information provided was helpful. If you still have questions, please don't hesitate to reach out to the community.
- v-priyankata4 months agoCommunity Support
Hi JasonKang
Hope everything’s going smoothly on your end. I wanted to check if the issue got sorted. if you have any other issues please reach community.