Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Anonymous
Not applicable

Get data from a web

Hi,

 

Could anybody help me to get the number of pages please? I think my problem is the condition to stop "each Table.IsEmpty(Page)," The call to this page has 1131 pages, and 1132 gives an error. Thanks.

 

let
GetData =
(PagePath as number) =>
let
Source =
Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&PagePath&"&s=100"
)
in
Source,
Source =
List.Generate(
() =>
[
page = 0,
Page = GetData(page)
],
each Table.IsEmpty(Page),
each
[
page = [page] + 1,
Page = GetData(page)
]
)
in
Source

1 ACCEPTED SOLUTION

@Anonymous , you might want to try a complete procedure like this if you insist on List.Generate().

let
    GetData = (PagePath as number) =>
        Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&Text.From(PagePath)&"&s=100")){[Name="fund"]}[Table],

    Source = List.Generate(
        () => [page = 1, Page = GetData(page)],
        each not (try [Page])[HasError],
        each [page = [page] + 1, Page = GetData(page)],
        each [Page]
    ),
    Dataset = Table.Combine(Source)
in
    Dataset

the condition to contiue the loop is a bit tricky,

each not (try [Page])[HasError]

Here's a detailed reference to the syntax of try statement in M language,

https://bengribaudo.com/blog/2020/01/15/4883/power-query-m-primer-part-15-error-handling


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

View solution in original post

5 REPLIES 5
CNENFRNL
Community Champion
Community Champion

@Anonymous , interestingly, I tried the assigned URL by chance, it returns

Screenshot 2020-12-08 221602.png

Thus, page parameter should range from 1 to 1131 rather than from 0.

I think the function should be defined as follows in order to scrape valid tables

 

GetData = (PagePath as number) =>
    let
        Source = Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p=" & Text.From(PagePath) & "&s=100")){[Name="fund"]}[Table]
    in
        Source,

 

 

In addition, List.Generate() can be replaced by List.Accumulate().

 

let
    GetData = (PagePath as number) =>
        let
            Source = Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p=" & Text.From(PagePath) & "&s=100")){[Name="fund"]}[Table]
        in
            Source,

    Source = List.Accumulate({1..1131}, {}, (s,c) => s & {GetData(c)}),
    Combination = Table.Combine(Source)
in
    Combination

 

Screenshot 2020-12-08 224224.png

 


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

Anonymous
Not applicable

Hi,

 

Thanks for this. Is that possible to work with List Generate? With List.Acumulate I will have to update {1..1131} as the website updates constantly with new content. That's why I sued ListGenerate, so I get the list of pages. So, I've updated the code with your suggestion, but I think the problem comes with the condition to stop [Page] = Null. 

thanks

 

let
GetData =
(PagePath as number) =>
let
Source = Xml.Tables(
Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&Text.From(PagePath)&"&s=100"),
{[Name="Fund"]}[Table]
)
in
Source,
Source =
List.Generate(
() =>
[
page = 1,
Page = GetData(page)
],
each [Page] = null,
each
[
page = [page] + 1,
Page = GetData(page)
]
)
in
Source

 

@Anonymous , you might want to try a complete procedure like this if you insist on List.Generate().

let
    GetData = (PagePath as number) =>
        Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/funds?p="&Text.From(PagePath)&"&s=100")){[Name="fund"]}[Table],

    Source = List.Generate(
        () => [page = 1, Page = GetData(page)],
        each not (try [Page])[HasError],
        each [page = [page] + 1, Page = GetData(page)],
        each [Page]
    ),
    Dataset = Table.Combine(Source)
in
    Dataset

the condition to contiue the loop is a bit tricky,

each not (try [Page])[HasError]

Here's a detailed reference to the syntax of try statement in M language,

https://bengribaudo.com/blog/2020/01/15/4883/power-query-m-primer-part-15-error-handling


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

Anonymous
Not applicable

Thanks. It worked!

However, I changed the website (bold) to grab data from other API (same website but different APIs depending on what info I want to grab). This also happens with other path. Both give "This table is empty". 

If this helps, this is the website https://gtr.ukri.org/resources/api.html, and these the endpoints http://gtr.ukri.org/gtr/api/examples 

 

Thanks.

 

let
GetData = (PagePath as number) =>
Xml.Tables(Web.Contents("https://gtr.ukri.org:443/gtr/api/projects?p="&Text.From(PagePath)&"&s=100")){[Name="fund"]}[Table],

Source = List.Generate(
() => [page = 1, Page = GetData(page)],
each not (try [Page])[HasError],
each [page = [page] + 1, Page = GetData(page)],
each [Page]
),
Dataset = Table.Combine(Source)
in
Dataset

CNENFRNL
Community Champion
Community Champion

Hi, @Anonymous 

As M language is extremely strick on data type, this expression

 

"https://gtr.ukri.org:443/gtr/api/funds?p="&PagePath&"&s=100"

 

throws error when a PagePath (as number) is passed to it.

You might want to try

 

https://gtr.ukri.org:443/gtr/api/funds?p=" & Text.From(PagePath) & "&s=100"

 

 


Thanks to the great efforts by MS engineers to simplify syntax of DAX! Most beginners are SUCCESSFULLY MISLED to think that they could easily master DAX; but it turns out that the intricacy of the most frequently used RANKX() is still way beyond their comprehension!

DAX is simple, but NOT EASY!

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Top Solution Authors
Top Kudoed Authors