Forum Discussion
Incremental Refresh - DateTime Partition Column Works in DEV but Initial Refresh Times Out in PROD
- 1 month ago
We have seen similar behavior with Power BI incremental refresh where Desktop succeeds but the first Service refresh fails or runs for hours.
A few key points I would check:
A DateTime partition column is supported
Unique timestamp values should not be an issue by themselves. Power BI should still create partitions based on the incremental refresh policy, not one partition per timestamp.The issue is likely execution/performance, not the DateTime column itself
Since Desktop refreshes in ~20 minutes but the Service runs for 4–5 hours and times out, I would look at Service-side execution differences: gateway latency, Premium capacity pressure, SQL timeout limits, and the actual SQL plans being used during refresh.Confirm query folding all the way back to SQL
The partition filter should fold into a clean SQL predicate similar to:WHERE DATE_TIME >= @RangeStart
AND DATE_TIME < @RangeEndAny casts, transformations, timezone handling, or non-foldable steps before the filter can make Service refresh much more expensive.
Test without Detect Data Changes
Temporarily disable Detect Data Changes and rerun the initial refresh. If it succeeds, the bottleneck may be the additional MAX(UPDATE_DATETIME) checks. If it still fails, focus on the partition query, gateway, SQL performance, or capacity.Review indexing
I would confirm SQL has an index supporting DATE_TIME, and review whether UPDATE_DATETIME also needs to be indexed or included based on the actual generated queries.My read is that the DateTime partition column is probably not the root cause. I would capture the SQL generated during the Service refresh, compare it against Desktop, and review the execution plans for the slowest partition queries.
We have seen similar behavior with Power BI incremental refresh where Desktop succeeds but the first Service refresh fails or runs for hours.
A few key points I would check:
A DateTime partition column is supported
Unique timestamp values should not be an issue by themselves. Power BI should still create partitions based on the incremental refresh policy, not one partition per timestamp.
The issue is likely execution/performance, not the DateTime column itself
Since Desktop refreshes in ~20 minutes but the Service runs for 4–5 hours and times out, I would look at Service-side execution differences: gateway latency, Premium capacity pressure, SQL timeout limits, and the actual SQL plans being used during refresh.
Confirm query folding all the way back to SQL
The partition filter should fold into a clean SQL predicate similar to:
WHERE DATE_TIME >= @RangeStart
AND DATE_TIME < @RangeEnd
Any casts, transformations, timezone handling, or non-foldable steps before the filter can make Service refresh much more expensive.
Test without Detect Data Changes
Temporarily disable Detect Data Changes and rerun the initial refresh. If it succeeds, the bottleneck may be the additional MAX(UPDATE_DATETIME) checks. If it still fails, focus on the partition query, gateway, SQL performance, or capacity.
Review indexing
I would confirm SQL has an index supporting DATE_TIME, and review whether UPDATE_DATETIME also needs to be indexed or included based on the actual generated queries.
My read is that the DateTime partition column is probably not the root cause. I would capture the SQL generated during the Service refresh, compare it against Desktop, and review the execution plans for the slowest partition queries.