Semantic Model Refresh Issue

[DPCCGF]

I have a model with an API connection that cycles through pages.  It is set at a 100 record per page selection.  The refresh works fine on the desktop but when published I get the following.  Is there a way around this?  Right now I up that 100 to like 10000 but I get timeouts.  So somehow I have to get around the below error.  TIA!

This dataset includes a dynamic data source. Since dynamic data sources aren't refreshed in the Power BI service, this dataset won't be refreshed. Learn more: https://aka.ms/dynamic-data-sources.

[TomMartens]

 Hey @DPCCGF ,

I recommend reading this article: https://hatfullofdata.blog/power-query-dynamic-data-source-and-web-contents/

Hopefully, this helps to tackle your challenge.

Regards,

Tom

[DPCCGF]

Hi Tom, any suggestions to the below?  I read the article but still having issues.

[DPCCGF]

Looks good, will this solution work even if I nee a token like shown below?  Credentials are Anonymous and Privacy is Public.

let

    Source = Json.Document(Web.Contents("https://XXXX.XXXXX.com/api/v1/" & "XXXXXXXXX.json" & "?_limit=100", [Headers=[#"Content-Type"="application/json", Authorization="Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"]])),

    totalPagesOrganizationPeople = (Source[count]/100),

    CreateList = List.Numbers(0,totalPagesOrganizationPeople,100),

    #"Converted to Table" = Table.FromList(CreateList, Splitter.SplitByNothing(), null, null, ExtraValues.Error),

    #"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", type text}}),

    #"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each Json.Document(Web.Contents("https://XXXX.XXXXX.com/api/v1/" & "organizations.json" & "?_limit=100" & "&_offset=" &[Column1], [Headers=[#"Content-Type"="application/json", Authorization="Token XXXXXXXXXXXXXXXXXXXXXXXXXXXX"]]))),

    #"Expanded Custom" = Table.ExpandRecordColumn(#"Added Custom", "Custom", {"results"}, {"Custom.results"}),

    #"Expanded Custom.results" = Table.ExpandListColumn(#"Expanded Custom", "Custom.results"),

    #"Expanded Custom.results1" = Table.ExpandRecordColumn(#"Expanded Custom.results", "Custom.results", {"url", "id", "organization", "person", "groups", "data", "alternate_id", "admin_notes", "documents", "status", "myuserpermissions", "assigned_programs", "created", "created_by", "modified", "modified_by"}, {"results.url", "results.id", "results.organization", "results.person", "results.groups", "results.data", "results.alternate_id", "results.admin_notes", "results.documents", "results.status", "results.myuserpermissions", "results.assigned_programs", "results.created", "results.created_by", "results.modified", "results.modified_by"})

in

    #"Expanded Custom.results1"