Forum Discussion
Could a Derived DateTime Partition Column Affect Incremental Refresh Performance?
Hi Team,
I noticed one key difference between our Missed Utterances report and our other reports that use Incremental Refresh.
For most of our reports, the partition column comes directly from the DATE_TIME dimension table:
dt.TXN_DATE AS IXN_SUBHOUR_DATE
This is obtained through a simple join using DATE_TIME_KEY.
However, in the Missed Utterances report, the partition column is derived as follows:
CAST(
START_TIME AT TIME ZONE 'UTC'
AT TIME ZONE 'Pacific Standard Time'
AS DATETIME) AS DATE_TIMEThe same Incremental Refresh configuration works successfully in our DEV environment, but in PROD the first Incremental Refresh runs for several hours and eventually fails with a SQL Execution Timeout, while a full refresh in Power BI Desktop completes successfully.
My question is:
Could using a derived DateTime expression (with AT TIME ZONE and CAST) as the Incremental Refresh partition column affect query folding, SQL optimization, or execution plans compared to using a DateTime column directly from the database?
Could this prevent SQL Server from using an Index Seek and instead cause an Index Scan or Table Scan during Incremental Refresh?
Is it considered a best practice to expose the converted DateTime as a physical column or view column in SQL Server rather than calculating it in the view used by Power BI?
I'd appreciate any guidance or recommendations based on your experience.
Thank you.
Hi manoj_0911 ,
Can you check the amount of data you are processing in the refresh ?The issue could be because of the volume of data being processed during the initial refresh.
A good first step would be to reduce the amount of data being loaded and test the refresh behavior.
What I usually do is create a parameter called LoadData (True/False) and use it in the Power Query layer as shown below:
Letsource....#"FirstNRows" = Table.FirstN(Source, 10),#"DataLoad" = if Loaddata then Source else #"FirstNRows"in#"DataLoad"
In Power BI Desktop, keep the parameter set to False by default to avoid encountering this issue during development.
Publish the file with parameter = False and do the first refresh so the partitions are created .Once the partitions are created, set the parameter to True in the Power BI Service and refresh each partition using SSMS.
If this response was helpful, please consider giving it a kudo and marking it as the correct solution—it helps other community members find answers more quickly.
4 Replies
- Natarajan_MSuper User
Hi manoj_0911 ,
Can you check the amount of data you are processing in the refresh ?The issue could be because of the volume of data being processed during the initial refresh.
A good first step would be to reduce the amount of data being loaded and test the refresh behavior.
What I usually do is create a parameter called LoadData (True/False) and use it in the Power Query layer as shown below:
Letsource....#"FirstNRows" = Table.FirstN(Source, 10),#"DataLoad" = if Loaddata then Source else #"FirstNRows"in#"DataLoad"
In Power BI Desktop, keep the parameter set to False by default to avoid encountering this issue during development.
Publish the file with parameter = False and do the first refresh so the partitions are created .Once the partitions are created, set the parameter to True in the Power BI Service and refresh each partition using SSMS.
If this response was helpful, please consider giving it a kudo and marking it as the correct solution—it helps other community members find answers more quickly.
- AnkitKukrejaSuper User
Hi! manoj_0911
Generally the amount of data available in Dev and prod differs. Your deployment to prod could be pulling in billion rows whereas in dev it could be in thousands.
I would then recommend to maybe reduce the timeframe used for incremental refresh maybe several years to 18 months or so, just an example. - v-saisrao-msftCommunity Support
Hi manoj_0911,
Have you had a chance to review the solution shared by Natarajan_M AnkitKukreja? If the issue persists, feel free to reply so we can help further.
Thank you.
- v-saisrao-msftCommunity Support
Hi manoj_0911,
Checking in to see if your issue has been resolved. let us know if you still need any assistance.
Thank you.