Join us at FabCon Atlanta from March 16 - 20, 2026, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.
Register now!The Power BI Data Visualization World Championships is back! Get ahead of the game and start preparing now! Learn more
Hi guys
I've posted a previous question on how to run a pagination loop in Pwer Query and have managed to get a working solution in Power BI Desktop:
https://community.powerbi.com/t5/Power-Query/PowerQuery-Rest-API-Pagination/m-p/1876297#M55640
I now face the issue that the datasource is dynamic and the Power BI Service does not support dynamic data sources. Any idea how I could change the query to be accepted by the Power BI Service for a scheduled refresh? I already removed the parameters (which I think are not the problem though) and incorporated it within the main query. But as the URL of the API changes for every page, the source remains dynamic.
I have the following function PageRunner:
(InputNumber)=>
let
APIKey = API_Key,
APPKey= APP_Key,
Page="&page="&Number.ToText(InputNumber),
Source = Json.Document(Web.Contents("https://api.SOFTWARE.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
#"Converted to Table" = Table.FromRecords({Source})
in
#"Converted to Table"
and this query:
let
MAXPages = let
APIKey = API_Key,
APPKey= APP_Key,
Page="&page="&Number.ToText(1),
Source = Json.Document(Web.Contents("https://api.SOFTWARE.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
#"Converted to Table" = Record.ToTable(Source),
Value = #"Converted to Table"{1}[Value],
pages1 = Value[pages]
in
pages1,
Source = List.Generate(
()=> [Page=1, Funct = PageRunner(1)],
each [Page]<=MAXPages,
each [Page=[Page]+1, Funct = PageRunner([Page]+1)]),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
#"Converted to Table"
Any hints? Thanks!!
Hi @Anonymous ,
To refresh a the dynamic data source in power bi service, you need to use RelativeParth and Query options in Web.Contents().
Source = Json.Document(
Web.Contents(
"https://api.SOFTWARE.com/api",
[
RelativePath = "v3/deals.json",
Query = [api_key="&APIKey&"&app_key="&APPKey&Page"]
]
)
)
Please refer:
Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Thanks @v-yingjl
It worked for most of the queries, I was able adjust the Fomulas I am calling in my iteration. But one creates an error: "Preview.Error: The type of the current preview value is too complex to display.
Details:"
I've tried to replicate it multiple times with changes back to the original code and doing the changes one by one, but for some reason this is the only query that causes the issue (even though there are other queries with more pages than this one).
To put it side by side, the old formula (without relative path) works just fine:
(InputNumber)=>
let
APIKey = API_Key,
APPKey= APP_Key,
Page="&page="&Number.ToText(InputNumber),
Source = Json.Document(Web.Contents(
"https://api.pipelinecrm.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
#"Converted to Table" = Table.FromRecords({Source})
in
#"Converted to Table"
But the new one...(PageRunnerDeals):
(InputNumber)=>
let
APIKey = API_Key,
APPKey= APP_Key,
Page=Number.ToText(InputNumber),
Source = Json.Document(Json.Document(
Web.Contents
("https://api.SOFTWARE.com/api",
[
RelativePath = "v3/deals.json",
Query = [api_key=APIKey,app_key=APPKey,page=Page]
]
)
)),
#"Converted to Table" = Table.FromRecords({Source})
in
#"Converted to Table"
Creates the error when called in the following query, as of step "= Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Funct"}, {"Page", "Funct"})":
let
MAXPages = let
APIKey = API_Key,
APPKey= APP_Key,
Page="&page="&Number.ToText(1),
Source = Json.Document(Web.Contents("https://api.pipelinecrm.com/api/v3/deals.json?api_key="&APIKey&"&app_key="&APPKey&Page)),
#"Converted to Table" = Record.ToTable(Source),
Value = #"Converted to Table"{1}[Value],
pages1 = Value[pages]
in
pages1,
Source = List.Generate(
()=> [Page=1, Funct = PageRunnerDeals(1)],
each [Page]<=MAXPages,
each [Page=[Page]+1, Funct = PageRunnerDeals([Page]+1)]),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Page", "Funct"}, {"Page", "Funct"}),
#"Expanded Column1.Funct" = Table.ExpandTableColumn(#"Expanded Column1", "Funct", {"entries", "pagination"}, {"entries", "pagination"}),
#"Expanded Column1.Funct.entries" = Table.ExpandListColumn(#"Expanded Column1.Funct", "entries")
in
#"Expanded Column1.Funct.entries"
Hi @Anonymous ,
Maybe consider these limitations about data source:
Dataset size limit - There's a 1-GB limit for datasets stored in Shared capacities in the Power BI service. If you need larger datasets, you can use Power BI Premium.
Distinct values in a column - When caching data in a Power BI dataset (sometimes called 'Import' mode), there is a 1,999,999,997 limit on the number of distinct values that can be stored in a column.
Row limit - When using DirectQuery, Power BI imposes a limit on the query results that are sent to your underlying data source. If the query sent to the data source returns more than one million rows, you see an error and the query fails. Your underlying data can still contain more than one million rows. You're unlikely to run into this limit as most reports aggregate the data into smaller sets of results.
Column limit - The maximum number of columns allowed in a dataset, across all tables in the dataset, is 16,000 columns. This limit applies to the Power BI service and to datasets used in Power BI Desktop. Power BI tracks the number of columns and tables in the dataset in this way, which means the maximum number of columns is 16,000 minus one for each table in the dataset.
Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
The Power BI Data Visualization World Championships is back! Get ahead of the game and start preparing now!