Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The Power BI Data Visualization World Championships is back! Get ahead of the game and start preparing now! Learn more

Reply
Anonymous
Not applicable

Refresh PowerQuery API Loop

Hi guys

 

I've posted a previous question on how to run a pagination loop in Pwer Query and have managed to get a working solution in Power BI Desktop:

https://community.powerbi.com/t5/Power-Query/PowerQuery-Rest-API-Pagination/m-p/1876297#M55640

 

I now face the issue that the datasource is dynamic and the Power BI Service does not support dynamic data sources. Any idea how I could change the query to be accepted by the Power BI Service for a scheduled refresh? I already removed the parameters (which I think are not the problem though) and incorporated it within the main query. But as the URL of the API changes for every page, the source remains dynamic.

 

I have the following function PageRunner:

 

(InputNumber)=>

let
    
    APIKey = API_Key,
    APPKey= APP_Key,
    Page="&page="&Number.ToText(InputNumber),
    Source = Json.Document(Web.Contents("https://api.SOFTWARE.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
    #"Converted to Table" = Table.FromRecords({Source})
    in
    #"Converted to Table"

 

  and this query:

 

let
    MAXPages = let
    
    APIKey = API_Key,
    APPKey= APP_Key,
    Page="&page="&Number.ToText(1),
    Source = Json.Document(Web.Contents("https://api.SOFTWARE.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
    #"Converted to Table" = Record.ToTable(Source),
    Value = #"Converted to Table"{1}[Value],
    pages1 = Value[pages]
in
    pages1,
    
    Source = List.Generate(
        ()=> [Page=1, Funct = PageRunner(1)],
        each [Page]<=MAXPages,
        each [Page=[Page]+1, Funct = PageRunner([Page]+1)]),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error)

in
    #"Converted to Table"

 

 

Any hints? Thanks!!

3 REPLIES 3
v-yingjl
Community Support
Community Support

Hi @Anonymous ,

To refresh a the dynamic data source in power bi service, you need to use RelativeParth and Query options in Web.Contents().

Source = Json.Document(
Web.Contents(
        "https://api.SOFTWARE.com/api",
        [
            RelativePath = "v3/deals.json",
            Query = [api_key="&APIKey&"&app_key="&APPKey&Page"]
        ]
    )
)

Please refer:

  1. Setting a scheduled refresh on a Dynamic Data Source in Power BI 
  2. Using The RelativePath And Query Options With Web.Contents() In Power Query And Power BI M Code 

 

Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Anonymous
Not applicable

Thanks @v-yingjl 

It worked for most of the queries, I was able adjust the Fomulas I am calling in my iteration. But one creates an error: "Preview.Error: The type of the current preview value is too complex to display.
Details:"
I've tried to replicate it multiple times with changes back to the original code and doing the changes one by one, but for some reason this is the only query that causes the issue (even though there are other queries with more pages than this one).

To put it side by side, the old formula (without relative path) works just fine:

 

(InputNumber)=>

let
    
    APIKey = API_Key,
    APPKey= APP_Key,
    Page="&page="&Number.ToText(InputNumber),
    Source = Json.Document(Web.Contents(
        "https://api.pipelinecrm.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
    #"Converted to Table" = Table.FromRecords({Source})
    in
#"Converted to Table"

 

 

 

But the new one...(PageRunnerDeals):

 

(InputNumber)=>

let
    
    APIKey = API_Key,
    APPKey= APP_Key,
    Page=Number.ToText(InputNumber),
    Source = Json.Document(Json.Document(
    Web.Contents
    ("https://api.SOFTWARE.com/api",
        [
            RelativePath = "v3/deals.json",
            Query = [api_key=APIKey,app_key=APPKey,page=Page]
        ]
    )
    )),
    #"Converted to Table" = Table.FromRecords({Source})
    in
#"Converted to Table"

 


Creates the error when called in the following query, as of step "= Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Funct"}, {"Page", "Funct"})":

 

let
    MAXPages = let
    
    APIKey = API_Key,
    APPKey= APP_Key,
    Page="&page="&Number.ToText(1),
    Source = Json.Document(Web.Contents("https://api.pipelinecrm.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
    #"Converted to Table" = Record.ToTable(Source),
    Value = #"Converted to Table"{1}[Value],
    pages1 = Value[pages]
in
    pages1,
    
    Source = List.Generate(
        ()=> [Page=1, Funct = PageRunnerDeals(1)],
        each [Page]<=MAXPages,
        each [Page=[Page]+1, Funct = PageRunnerDeals([Page]+1)]),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Funct"}, {"Page", "Funct"}),
    #"Expanded Column1.Funct" = Table.ExpandTableColumn(#"Expanded Column1", "Funct", {"entries", "pagination"}, {"entries", "pagination"}),
    #"Expanded Column1.Funct.entries" = Table.ExpandListColumn(#"Expanded Column1.Funct", "entries")
in
    #"Expanded Column1.Funct.entries"

 




Hi @Anonymous ,

Maybe consider these limitations about data source:

  • Dataset size limit - There's a 1-GB limit for datasets stored in Shared capacities in the Power BI service. If you need larger datasets, you can use Power BI Premium.

  • Distinct values in a column - When caching data in a Power BI dataset (sometimes called 'Import' mode), there is a 1,999,999,997 limit on the number of distinct values that can be stored in a column.

  • Row limit - When using DirectQuery, Power BI imposes a limit on the query results that are sent to your underlying data source. If the query sent to the data source returns more than one million rows, you see an error and the query fails. Your underlying data can still contain more than one million rows. You're unlikely to run into this limit as most reports aggregate the data into smaller sets of results.

  • Column limit - The maximum number of columns allowed in a dataset, across all tables in the dataset, is 16,000 columns. This limit applies to the Power BI service and to datasets used in Power BI Desktop. Power BI tracks the number of columns and tables in the dataset in this way, which means the maximum number of columns is 16,000 minus one for each table in the dataset.

 

Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
Power BI DataViz World Championships

Power BI Dataviz World Championships

The Power BI Data Visualization World Championships is back! Get ahead of the game and start preparing now!

December 2025 Power BI Update Carousel

Power BI Monthly Update - December 2025

Check out the December 2025 Power BI Holiday Recap!

FabCon Atlanta 2026 carousel

FabCon Atlanta 2026

Join us at FabCon Atlanta, March 16-20, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.