Scraping of Data from Multiple Web Pages with no page # in the URL

[soumyaiyer]

Hello,

I am trying to extract all the pages of data from the URL (https://apps2.xxxxxxxx.net)

I am able to bring in the first 25 with web connector and extract HTML but I need all the pages. The URL does not change as you move through the pages on the site.

The advanced editor requires page number, but the URL I have does not show page numbers. The webpage has only the arrow option to view the next page.

Any help would be appreciated.

Thanks

[MarkLaf · Accepted solution]

Hit F12 in your browser, this brings up web dev tools in most browsers. You can then inspect the html and other components of the page for clues as to how to grab the underlying data.

Your best bet, I'd guess, would be to look at the network traffic and look out for http api calls when you click the Next button. You may have to inspect the headers and body in the case of POSTs.

There is a chance that it's just impossible with straight querying, depending on how it's built. You may have to look into rpa with Power Automate or some other product.

[soumyaiyer]

Below is what I see in the query advanced editor, this helps only to view the first 1-25

let
Source = Web.BrowserContents("https://apps2.xxxx.net/customerservice/campaign/?tag=xxx_inquiries"),
#"Extracted Table From Html" = Html.Table(Source, {{"Column1", "DIV.gridless > TABLE > * > TR > :nth-child(1)"}, {"Column2", "DIV.gridless > TABLE > * > TR > :nth-child(2)"}, {"Column3", "DIV.gridless > TABLE > * > TR > :nth-child(3)"}, {"Column4", "DIV.gridless > TABLE > * > TR > :nth-child(4)"}, {"Column5", "DIV.gridless > TABLE > * > TR > :nth-child(5)"}, {"Column6", "DIV.gridless > TABLE > * > TR > :nth-child(6)"}, {"Column7", "DIV.gridless > TABLE > * > TR > :nth-child(7)"}}, [RowSelector="DIV.gridless > TABLE > * > TR"]),
#"Promoted Headers" = Table.PromoteHeaders(#"Extracted Table From Html", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"", type text}, {"ID", Int64.Type}, {"Request Type", type text}, {"Customers", type text}, {"Accounts", Int64.Type}, {"Agent", type text}, {"Queued Since", type datetime}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{""}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns",{{"ID", type text}})
in
#"Changed Type1"

[MarkLaf · Accepted solution]

Hit F12 in your browser, this brings up web dev tools in most browsers. You can then inspect the html and other components of the page for clues as to how to grab the underlying data.

Your best bet, I'd guess, would be to look at the network traffic and look out for http api calls when you click the Next button. You may have to inspect the headers and body in the case of POSTs.

There is a chance that it's just impossible with straight querying, depending on how it's built. You may have to look into rpa with Power Automate or some other product.

[soumyaiyer]

Tried the Power Automate route the  company has restrictions in providing access to the desktop version . Can I web scrap in the Power Automate cloud version?

[MarkLaf · Accepted solution]

I don't believe so. In case the restrictions are due to security/whatever and not cost or complexity, you could run it off of a virtual machine: https://learn.microsoft.com/en-us/power-automate/desktop-flows/hosted-rpa-overview

It may be worth asking in the Power Automate forums if that's the route you are thinking of going. You're more likely to find folks with the experience to speak to the potential options/considerations.