Forum Discussion
OData Incremental refresh compatible with schedule refresh
- 3 years ago
Check out the second approach in this article that shows an example of using List.Generate to recursively use a nextlink type url to page through responses until the last one. The key step is the one called datalist.
Updated – Get SharePoint List Data … Fast – Hoosier BI
Pat
Check out the second approach in this article that shows an example of using List.Generate to recursively use a nextlink type url to page through responses until the last one. The key step is the one called datalist.
Updated – Get SharePoint List Data … Fast – Hoosier BI
Pat
Thank you Pat !
Here is my code if someone else needs it.
I've splitted the query in two because in the transformation step (the one that convert the list of response to a table) was failing in Power BI Service. Probably due to empty response. Error was saying "Column1" (the field where the list is stored) did not exists
So in order to setup incremental refresh on the final table, I kinda tricked PowerBI by filtering the final result using RangeStart and RangeEnd, but I think it could be more effiencient to just use a dummy step
"myRessource Raw" M code:
(Get all the API pages using paginate_field)
let
base_url = "https://myurl.com/odata/1.0/" ,
ressource_path = "myRessources",
relative_path = technical_workspace & "/" & ressource_path,
date_field = "createdAt",
paginate_field = #"@odata.nextLink",
InitialWebCall = Json.Document(
Web.Contents(
base_url ,
[
RelativePath = relative_path,
Query=[#"$filter" = date_field & " ge '" & DateTime.ToText(RangeStart, "yyyy-MM-dd") & "' and " & date_field & " lt '" & DateTime.ToText(RangeEnd, "yyyy-MM-dd") & "'" ]
]
)
),
Datalist = List.Generate(
() => InitialWebCall,
each List.Count([value]) > 0,
each
try
Json.Document(
Web.Contents(base_url, [
RelativePath = relative_path & Text.AfterDelimiter([paginate_field], ressource_path)
]))
otherwise [value = {}],
each [value])
in
Datalist
"myRessource" code :
(Convert the list of response to table data)
let
Source = #"myRessources Raw",
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandListColumn(#"Converted to Table", "Column1"), //this is probably this step that was failing in Power BI Service
#"Column1 Expand Record" = Table.ExpandRecordColumn(#"Expanded Column1", "Column1", {"_id", "attributeName", "event_id", "attributeFilter", "attributeValue", "createdAt"}, {"_id", "attributeName", "event_id", "attributeFilter", "attributeValue", "createdAt"}),
#"Modified Type" = Table.TransformColumnTypes(#"Column1 Expand Record",{{"createdAt", type datetime}}),
#"Filter Lines" = Table.SelectRows(#"Modified Type", each [receivedAt] >= RangeStart and [receivedAt] < RangeEnd) //step added to tricks Power BI in order to be able to setup incremental refresh
in
#"Filter Lines"
This way it works really well, I can schedule refresh in Power BI Service and incremental refresh seems to work just fine
- tanja921 year agoNew Member
Hi Ronan,
I have the same issue. Could you explain your solution to me in more detail? I know it was some time ago, but if you're still able to explain it, it would be really helpful!