Web content data extraction

[Maheshwari]

Hi All iam pulling data using web url. the web url shows data for only first 100 rows in one page.if we need to get further data need to click buttons 2,3,4..done using href dopostback command.but the main url remains same for all pages.so i get only first page data.is there any way to get all page data?

[v-stephen-msft]

Hi @Maheshwari ,

Please kindly refer to the blog:

https://community.powerbi.com/t5/Community-Blog/Batch-crawling-web-page-data/ba-p/1715990

In the blog, the author used this web site as an example, follow the steps below

1.Analysis of the structure of the website
2.Pull out the first page of data using PowerBI
3.Set the custom function based on the page number parameter
4.Batch calls to custom functions
5.Batch change to column names

Best Regards,

Stephen Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Maheshwari]

Hi @v-stephen-msft ,thanks for sharing...but the web url remains same for all page numbers..it uses some href dopostback command..even looked into page source code...not sure how to get different links for each page

[v-stephen-msft]

Hi @Maheshwari ,

Can you share the web url with me?

Best Regards,

Stephen Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[mahoneypat]

That will depend on how the web page is structured.  If you see a url that contains a page value in it, you can make a list of values and then concatenate it into the url and make a web call for each page.  If not, may not be possible.

Pat

[Expresso]

I have the same problem, how to do that?