Forum Discussion
Service refresh after pagination implementation with web data source
- 6 years ago
Hi Anonymous ,
It depends on the detail situation, you can upload it to Power BI Service and verify if it can configure scheduled refresh, we think if it does not work, the issue maybe url is dynamic , you can try to use following query which use query option to replace combine url manually, you can also use relative path, please refer to following document about query option and relative path: https://docs.microsoft.com/en-us/powerquery-m/web-contents
let BaseUrl = "https://XXXX/report?", Password = "XXXX", Report = "sales", Fields = "", EntitiesPerPage = 1000, GetJson = (QParm) => let Options = [Headers=[ #"Authorization" = "Bearer " & Token ], Query = QParm], RawData = Web.Contents(BaseUrl, Options), Json = Json.Document(RawData) in Json, GetEntityCount = () => let QParm = [pw=Password,report=Report,count="true",per_page=Text.From(EntitiesPerPage)], Json = GetJson(QParm), Count = Json[#"total_pages"] in Count, GetPage = (Index) => let Page = "page=" & Text.From(Index), QParm = [pw=Password,report=Report,page=Page,per_page=Text.From(EntitiesPerPage),fields=Fields], Json = GetJson(QParm), Value = Json[#"data"] in Value, PageCount = GetEntityCount(), PageIndices = { 1 .. PageCount }, Pages = List.Transform(PageIndices, each GetPage(_)), Entities = List.Union(Pages), Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error) in TablePlease also consider the limitation of you api, it may block the request if they are too frequency or reach the api limitation (such as number limitation per day)
Best regards,
Hi Anonymous ,
It depends on the detail situation, you can upload it to Power BI Service and verify if it can configure scheduled refresh, we think if it does not work, the issue maybe url is dynamic , you can try to use following query which use query option to replace combine url manually, you can also use relative path, please refer to following document about query option and relative path: https://docs.microsoft.com/en-us/powerquery-m/web-contents
let
BaseUrl = "https://XXXX/report?",
Password = "XXXX",
Report = "sales",
Fields = "",
EntitiesPerPage = 1000,
GetJson = (QParm) =>
let
Options = [Headers=[ #"Authorization" = "Bearer " & Token ], Query = QParm],
RawData = Web.Contents(BaseUrl, Options),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () =>
let
QParm = [pw=Password,report=Report,count="true",per_page=Text.From(EntitiesPerPage)],
Json = GetJson(QParm),
Count = Json[#"total_pages"]
in
Count,
GetPage = (Index) =>
let
Page = "page=" & Text.From(Index),
QParm = [pw=Password,report=Report,page=Page,per_page=Text.From(EntitiesPerPage),fields=Fields],
Json = GetJson(QParm),
Value = Json[#"data"]
in
Value,
PageCount = GetEntityCount(),
PageIndices = { 1 .. PageCount },
Pages = List.Transform(PageIndices, each GetPage(_)),
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
Table
Please also consider the limitation of you api, it may block the request if they are too frequency or reach the api limitation (such as number limitation per day)
Best regards,
v-lid-msft when using your solution, with fields"..." the format we are sending, we are not getting all the fields
- v-lid-msft6 years agoCommunity Support
Hi Anonymous ,
Could you please try to verify what is difference between the request posted by the new query and origin? We also use the Relative Path option as fololwing, it should act as the same as combine url in origin query when post the request
let BaseUrl = "https://XXXX/report?", Password = "XXXX", Report = "sales", Fields = "", EntitiesPerPage = 1000, GetJson = (RParm) => let Options = [Headers=[ #"Authorization" = "Bearer " & Token ], RelativePath = RParm, RawData = Web.Contents(BaseUrl, Options), Json = Json.Document(RawData) in Json, GetEntityCount = () => let RParm = "pw="& Password & "&report=" & Report &"&count=true&per_page="& Text.From(EntitiesPerPage), Json = GetJson(RParm), Count = Json[#"total_pages"] in Count, GetPage = (Index) => let Page = "page=" & Text.From(Index), RParm = "pw="& Password & "&report=" & Report & "&page=" & Page & "&per_page=" & PerPage & "&fields=" & Fields, Json = GetJson(RParm), Value = Json[#"data"] in Value, PageCount = GetEntityCount(), PageIndices = { 1 .. PageCount }, Pages = List.Transform(PageIndices, each GetPage(_)), Entities = List.Union(Pages), Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error) in Table
Best regards, - v-lid-msft6 years agoCommunity Support
Hi Anonymous ,
How about the result after you follow the suggestions mentioned in my previous post?Could you please provide more details about it If it doesn't meet your requirement?
Best regards,- Anonymous6 years agoNot applicable
The issue was in the variables i put in the link. I was formatting them with %20 for spaces and i should not have. So your solution worked!