Forum Discussion
Use a table value as a filter parameter when Refreshing the same table
First, make a new step in your table query like:
MaxDate = List.Max(#"Table Name"[DateTime])
Now you have your filter value, which is the latest datetime value in the column.
Now let's say that prior to us adding this step, your final step was named "Table":
NewParam = Table.SelectRows(Table, each Date.From(DateTime.LocalNow()) >= MaxDate)
--Nate
- Anonymous5 years agoNot applicable
Thanks for the suggestion Nate, although I think my connector is slightly more complicated as the Web Service is also wrapped in Pagination logic:
This is the snipped from the Connector that calls the XML Web Service, with your suggested code added:
XMLWebService = (hostUrl as text, gridCode as text, optional filterField as text, optional RangeStart as datetime, optional RangeEnd as datetime) => let hostUrl = hostUrl, gridCodeString = """" & gridCode & """", filterFieldString = """" & filterField & """", filterFrom = if RangeStart = null then """1900-01-01""" else """" & RangeStart & """", filterTo = if RangeEnd = null then """""" else """" & RangeEnd & """", XMLUserName = Extension.CurrentCredential()[Username], XMLUserPassword = Extension.CurrentCredential()[Password], // GetXML = GetXMLData(hostUrl,XMLUserName, XMLUserPassword, gridCodeString,1, filterFieldString, RangeStart, RangeEnd), Pagination = List.Skip(List.Generate( () => [IsMore = null, Last_Key = 0, Counter = 0], // Start Value each [IsMore] <> false,// Whilst this is true, keep going each [GetXML = GetXMLData(hostUrl,XMLUserName, XMLUserPassword, gridCodeString,Last_Key, filterFieldString,MaxDate, RangeEnd), Last_Key = try [GetXML][#"cursorPosition"] otherwise 0, IsMore = if [Counter] < 1 then null else [GetXML][#"hasMore"], Counter = [Counter]+1, splitList = Table.FromRows(GetXML[rows], GetXML[fieldNameList]), #"Transform Types" = Table.TransformColumnTypes(splitList, List.Zip( {GetXML[fieldNameList], GetXML[fieldDataTypes] } )), MaxDate = List.Max(#"Transform Types"[last_refreshed]) ] ,each [#"Transform Types"] // selector ) ,1) , Custom = Table.Combine(Pagination) // output = GetXML in Custom ;This on its own works fine. The RangeStart and RangeEnd parameters are there from my trial and error and will likely be removed when this is all working.
Where i'm struggling with your suggestion is the the last step which creates the table is in the Pagination Logic, and perhaps i'm looking at this too much through my SQL glasses where you have to specify which table to get the value from and SQL evaluates in the order its written.
I added the MaxDate which is after the #"Transform Types" (this step takes the table of data and forces the data types to match the incoming data types as PowerBI, surprisingly, gets them all wrong.)
I then referenced the MaxDate further up in the Pagination item > GetXML function parameters (the GetXML function is the full XML Web Service code which is where the filter is used.
When trying to refresh in PowerBI, i'm getting a cyclic reference error (in this image, 'Property' is the name of the table being refreshed).
In my mind I sort of know what it means (i'm guessing its like a circular reference error in Excel or SQL), but as i'm new to Power Query/Power BI, i'm not sure where to go from here.
Ultimately, in my test environment, I have already loaded the 'Property' table and the 'last_refreshed' is available.
what it needs to to is:
- Click Refresh (or scheduled refresh)
- Get the Max date from the data that is already loaded (in this case, 24/07/2021 4:28:52 AM)
- Pass that date to the Web Service so it will only get new records from the Source system where the data was updated on or after 24/07/2021 4:28:52 AM