Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The Power BI DataViz World Championships are on! With four chances to enter, you could win a spot in the LIVE Grand Finale in Las Vegas. Show off your skills.

Reply
Anonymous
Not applicable

Get data from a web

Hi,

 

Could anybody help me to get the number of pages please? I think my problem is the condition to stop "each Table.IsEmpty(Page)," The call to this page has 1131 pages, and 1132 gives an error. Thanks.

 

let
GetData =
(PagePath as number) =>
let
Source =
Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&PagePath&"&s=100"
)
in
Source,
Source =
List.Generate(
() =>
[
page = 0,
Page = GetData(page)
],
each Table.IsEmpty(Page),
each
[
page = [page] + 1,
Page = GetData(page)
]
)
in
Source

1 ACCEPTED SOLUTION

@Anonymous , you might want to try a complete procedure like this if you insist on List.Generate().

let
    GetData = (PagePath as number) =>
        Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&Text.From(PagePath)&"&s=100")){[Name="fund"]}[Table],

    Source = List.Generate(
        () => [page = 1, Page = GetData(page)],
        each not (try [Page])[HasError],
        each [page = [page] + 1, Page = GetData(page)],
        each [Page]
    ),
    Dataset = Table.Combine(Source)
in
    Dataset

the condition to contiue the loop is a bit tricky,

each not (try [Page])[HasError]

Here's a detailed reference to the syntax of try statement in M language,

https://bengribaudo.com/blog/2020/01/15/4883/power-query-m-primer-part-15-error-handling


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

View solution in original post

5 REPLIES 5
CNENFRNL
Community Champion
Community Champion

@Anonymous , interestingly, I tried the assigned URL by chance, it returns

Screenshot 2020-12-08 221602.png

Thus, page parameter should range from 1 to 1131 rather than from 0.

I think the function should be defined as follows in order to scrape valid tables

 

GetData = (PagePath as number) =>
    let
        Source = Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p=" & Text.From(PagePath) & "&s=100")){[Name="fund"]}[Table]
    in
        Source,

 

 

In addition, List.Generate() can be replaced by List.Accumulate().

 

let
    GetData = (PagePath as number) =>
        let
            Source = Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p=" & Text.From(PagePath) & "&s=100")){[Name="fund"]}[Table]
        in
            Source,

    Source = List.Accumulate({1..1131}, {}, (s,c) => s & {GetData(c)}),
    Combination = Table.Combine(Source)
in
    Combination

 

Screenshot 2020-12-08 224224.png

 


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

Anonymous
Not applicable

Hi,

 

Thanks for this. Is that possible to work with List Generate? With List.Acumulate I will have to update {1..1131} as the website updates constantly with new content. That's why I sued ListGenerate, so I get the list of pages. So, I've updated the code with your suggestion, but I think the problem comes with the condition to stop [Page] = Null. 

thanks

 

let
GetData =
(PagePath as number) =>
let
Source = Xml.Tables(
Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&Text.From(PagePath)&"&s=100"),
{[Name="Fund"]}[Table]
)
in
Source,
Source =
List.Generate(
() =>
[
page = 1,
Page = GetData(page)
],
each [Page] = null,
each
[
page = [page] + 1,
Page = GetData(page)
]
)
in
Source

 

@Anonymous , you might want to try a complete procedure like this if you insist on List.Generate().

let
    GetData = (PagePath as number) =>
        Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&Text.From(PagePath)&"&s=100")){[Name="fund"]}[Table],

    Source = List.Generate(
        () => [page = 1, Page = GetData(page)],
        each not (try [Page])[HasError],
        each [page = [page] + 1, Page = GetData(page)],
        each [Page]
    ),
    Dataset = Table.Combine(Source)
in
    Dataset

the condition to contiue the loop is a bit tricky,

each not (try [Page])[HasError]

Here's a detailed reference to the syntax of try statement in M language,

https://bengribaudo.com/blog/2020/01/15/4883/power-query-m-primer-part-15-error-handling


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

Anonymous
Not applicable

Thanks. It worked!

However, I changed the website (bold) to grab data from other API (same website but different APIs depending on what info I want to grab). This also happens with other path. Both give "This table is empty". 

If this helps, this is the website https://gtr.ukri.org/resources/api.html, and these the endpoints http://gtr.ukri.org/gtr/api/examples 

 

Thanks.

 

let
GetData = (PagePath as number) =>
Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/projects?p="&Text.From(PagePath)&"&s=100")){[Name="fund"]}[Table],

Source = List.Generate(
() => [page = 1, Page = GetData(page)],
each not (try [Page])[HasError],
each [page = [page] + 1, Page = GetData(page)],
each [Page]
),
Dataset = Table.Combine(Source)
in
Dataset

CNENFRNL
Community Champion
Community Champion

Hi, @Anonymous 

As M language is extremely strick on data type, this expression

 

"https://gtr.ukri.org:443/gtr/api/funds?p="&PagePath&"&s=100"

 

throws error when a PagePath (as number) is passed to it.

You might want to try

 

https://gtr.ukri.org:443/gtr/api/funds?p=" & Text.From(PagePath) & "&s=100"

 

 


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

Helpful resources

Announcements
Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

Jan NL Carousel

Fabric Community Update - January 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors
Top Kudoed Authors