Forum Discussion

jdc2019's avatar
jdc2019
New Member
1 year ago
Solved

Incremental Refresh with Elastic Tables and Data Detection

I have a scenario where I am trying to implement incremental refresh on an elastic table using the powerbi service and a semantic model.  I'm using the ExecuteCosmosSqlQuery endpoint, which allows me to run a NoSQL query that folds until i expand it.  The query returns a createdon and modifiedon date in the format "yyyy-MM-ddTHH:mm:ssZ".  After LOTS of research and work with copilot, i verified that i can convert my RangeStart and RangeEnd Date/Time parameters to strings and even pick up the last valid refresh timestamp with a function like this:

FormatRangeStart = DateTime.ToText(RangeStart, "yyyy-MM-ddTHH:mm:ssZ"),
FormatRangeEnd = DateTime.ToText(RangeEnd, "yyyy-MM-ddTHH:mm:ssZ"),
FormatLastRefresh = LastRefreshTimeStamp("ElasticCostCollection")

 

My query uses paging and cookies, and all of this works.

JSONPage = (PageCookie) =>
// Make a new record based on the JSON read of the ExecuteCosmosSQLQuery function
// the API returns one page of records. If there is another page
// then the Paging Cookie will have a value and the HasMore boolean will be true.
// If HasMore is false, then this is the last read so we set a boolean called
// LastAPICall to true
let
NewRecord =
Json.Document(
Web.Contents(
"https://" & Org & EnvironmentZone,
[
RelativePath = "api/data/v9.2/ExecuteCosmosSqlQuery(QueryText=@p1,EntityLogicalName=@p2,PageSize=5000,PagingCookie=@p3)",
Query = [
#"@p1" = "'select " &
"c.props.tms_ecostcollection_controlaccount As tms_ecostcollection_controlaccount, " &
"c.props.tms_ecostcollection_elasticplanning As tms_ecostcollection_elasticplanning, " &
"c.props.tms_ecostcollection_project, " &
"c.props.tms_ecostcollection_ratestack, " &
"c.props.tms_ecostcollection_resource, " &
"c.props.tms_ecostcollection_resultunit, " &
"c.props.tms_ecostcollection_scenario, " &
"c.props.tms_ecostcollection_workpackage, " &
"c.props.tms_elasticcostcollectiondescription, " &
"c.props.tms_elasticcostcollectioneffectivedate, " &
"c.props.tms_elasticcostcollectionelementofcost, " &
"c.props.tms_elasticcostcollectionid, " &
"c.props.tms_elasticrecordid, " &
"c.props.tms_resultunitvalue, " &
"c.props.tms_transactiontype, " &
"c.props.tms_elasticcostcollectionaccountingcode, " &
"c.props.tms_elasticcostcollection_bcr, " &
"c.props.tms_ecostcollection_replanbcr, " &
"c.props.tms_ecostcollection_status, " &
"c.props.tms_ecostcollection_statusreason, " &
"c.props.tms_ecostcollection_ratesheet, " &
"c.props.tms_ecostcollection_indirectpool, " &
"c.props.tms_elasticcostcollectionappliedpoolrate, " &
"c.props.tms_elasticcostcollection_directrate, " &
"c.props.tms_elasticcostcollectionnote, " &
"c.props.partitionid, " &
"c.props.modifiedon, " &
"c.props.createdon " &
"from c " &
"WHERE c.props.createdon >= " & """" & FormatRangeStart & """" &
" AND c.props.createdon < " & """" & FormatRangeEnd & """" &
" AND c.props.modifiedon > " & """" & FormatLastRefresh & """" &
"'",
#"@p2" = "'tms_elasticcostcollection'",
#"@p3" = "'" & PageCookie & "'"
],
Headers = [
Authorization = AuthToken
]
]
)
),
Output =
Record.AddField(
NewRecord,
"LastAPICall", not Record.Field(NewRecord, "HasMore")
)
in
Output,
// In the list generate function, we run the function to pull a page from the API, and then test
// that function. Whether it is the last page or not, we accept the record into the list
// if this read was the last page, we do not call the JSONPage function again, but simply
// create an empty record with both flags set to false so the list generate function will terminate
PageRecords = List.Generate(
() => JSONPage(""),
each _[HasMore] or _[LastAPICall],
each
if not _[LastAPICall] then
JSONPage(_[PagingCookie])
else
Record.FromList (
{false,false},
{"HasMore","LastAPICall"}
)
),

 

It's not possible to convert the modifiedon date to a date/time field prior to the break in query folding, because the expansion of the list records to access columns breaks folding.  Copilot indicated that I should be able to add the last refresh date check in the where clause of the query (while it's still folding) and that should work, with the powerbi service retaining the records that didn't change (where the modified on date was prior to the last refresh timestamp). 

 

What happens:  My query seems to correctly honor the  partitions with the Range parameters, but if I run the refresh a second time, all of the records that filter out by the modified on date disappear - the service is not retaining previously loaded records in the incremental refresh period.  Does anyone know a way to actually use data detection on a field that represents a date/time value but returns as a string?  Any help would be greatly appreciated!

 

  • Hi jdc2019,

    Thank you for your follow-up! You’ve raised an excellent question regarding how Power BI’s incremental refresh engine determines partition updates.

    Yes, the Power BI refresh engine evaluates partitions using the modifiedon field, but for it to work optimally, the query must still be folding at the point where modifiedon is treated as a DateTime.

    • The refresh engine assesses partitions using the RangeStart and RangeEnd parameters.
    • If Power BI can apply the modifiedon filter while query folding is still intact, it will evaluate which partitions need refreshing based on whether the modifiedon value is greater than the last refresh timestamp.
    • If query folding breaks before modifiedon is converted to a DateTime, Power BI cannot efficiently determine changes at the partition level, which may lead to unexpected behavior like full partition replacement.

    I trust this information proves useful. If it does, kindly Accept it as a solution and give it a 'Kudos' to help others locate it easily.
    Thank you.

8 Replies

  • but if I run the refresh a second time, all of the records that filter out by the modified on date disappear - the service is not retaining previously loaded records in the incremental refresh period. 

     

    The Change detection impacts the entire partition. Power BI Incremental Refresh is incapable of replacing/updating individual rows. So most likely your Last Modified Date is impacting more than just the "hot" partition(s)

  • v-ssriganesh's avatar
    v-ssriganesh
    Community Support

    Hi jdc2019,

    Thanks for sharing your scenario in the community! Also, great insights from lbendlin Power BI’s Incremental Refresh indeed operates at the partition level, meaning that when changes occur within a partition, the entire partition is refreshed rather than individual rows being updated.

    Since your query filters records based on modifiedon > LastRefreshTimestamp, it might be causing entire partitions to be excluded from the dataset on subsequent refreshes. When Power BI refreshes a partition, it does not merge with previously loaded records it replaces the entire partition with the new filtered dataset. This could explain why records that did not meet the modified date condition are disappearing.

    Consider the below Steps:

    • Please ensure modifiedon is correctly recognized as a DateTime before query folding breaks.
    • Remove the filter on modifiedon in your query and let Power BI handle change detection at the partition level. Instead of pre-filtering in the query, ensure that incremental refresh settings are correctly applied in Power BI Service.
    • Ensure that the incremental refresh window aligns with your expected data retention. Power BI retains only the most recent partitions based on the defined archival period.

    If this helps, then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.
    Thank you.

    • jdc2019's avatar
      jdc2019
      New Member

      First - Thank you Ibendlin for the information about data change detection and refresh working only at the partition level.  That is not clear from the Microsoft documentation or from any queries to CoPilot.  As for the proposed solution, it is not possible to get the modifiedon date in date/time format when extracting the field from a cosmos endpoint.  The act of expanding the query to access the columns breaks folding.  I needed to understand the algorithm to address this, so is it true that the incremental refresh engine is sampling each partition using the modifiedon date field to determine whether that date is greater than the last refresh date?  If so, then I can't use the built in refresh engine to test since I can't access the date/time value of modifiedon prior to the break in query folding.  I can probably mimic this behavior by manually partitioning and running the algorithm myself with the powerbi api.  I'm using that already to find the last refresh date.  Can you confirm that the refresh engine is sampling each partition, and the query must fold at the point where the modifiedon date is a date/time value in order for it to optimize the refresh?

      • v-ssriganesh's avatar
        v-ssriganesh
        Community Support

        Hi jdc2019,

        Thank you for your follow-up! You’ve raised an excellent question regarding how Power BI’s incremental refresh engine determines partition updates.

        Yes, the Power BI refresh engine evaluates partitions using the modifiedon field, but for it to work optimally, the query must still be folding at the point where modifiedon is treated as a DateTime.

        • The refresh engine assesses partitions using the RangeStart and RangeEnd parameters.
        • If Power BI can apply the modifiedon filter while query folding is still intact, it will evaluate which partitions need refreshing based on whether the modifiedon value is greater than the last refresh timestamp.
        • If query folding breaks before modifiedon is converted to a DateTime, Power BI cannot efficiently determine changes at the partition level, which may lead to unexpected behavior like full partition replacement.

        I trust this information proves useful. If it does, kindly Accept it as a solution and give it a 'Kudos' to help others locate it easily.
        Thank you.