Forum Discussion

manoj_0911's avatar
manoj_0911
Kudo Commander
2 months ago
Solved

Incremental Refresh - DateTime Partition Column Works in DEV but Initial Refresh Times Out in PROD

  Hi Team, I am investigating an Incremental Refresh issue with our Missed Utterances report and would appreciate your guidance. Current Configuration Partition column: DATE_TIME (DateTime with...
  • babybots's avatar
    1 month ago

    We have seen similar behavior with Power BI incremental refresh where Desktop succeeds but the first Service refresh fails or runs for hours.

    A few key points I would check:

     

    A DateTime partition column is supported
    Unique timestamp values should not be an issue by themselves. Power BI should still create partitions based on the incremental refresh policy, not one partition per timestamp.

     

    The issue is likely execution/performance, not the DateTime column itself
    Since Desktop refreshes in ~20 minutes but the Service runs for 4–5 hours and times out, I would look at Service-side execution differences: gateway latency, Premium capacity pressure, SQL timeout limits, and the actual SQL plans being used during refresh.

     

    Confirm query folding all the way back to SQL
    The partition filter should fold into a clean SQL predicate similar to:

     
    WHERE DATE_TIME >= @RangeStart
    AND DATE_TIME < @RangeEnd
     

    Any casts, transformations, timezone handling, or non-foldable steps before the filter can make Service refresh much more expensive.

     

    Test without Detect Data Changes
    Temporarily disable Detect Data Changes and rerun the initial refresh. If it succeeds, the bottleneck may be the additional MAX(UPDATE_DATETIME) checks. If it still fails, focus on the partition query, gateway, SQL performance, or capacity.

     

    Review indexing
    I would confirm SQL has an index supporting DATE_TIME, and review whether UPDATE_DATETIME also needs to be indexed or included based on the actual generated queries.

     

    My read is that the DateTime partition column is probably not the root cause. I would capture the SQL generated during the Service refresh, compare it against Desktop, and review the execution plans for the slowest partition queries.