Web scrapping full data into a table - including load more

[Anonymous]

Hello friends,

am trying to do scrapping of a retail website and when I do the scrapping I can get only the data seen in the screen for example if there are only 10 items on the screen only those are shown in the powerbi table. But when we click on load more on the web page more items are displayed but how do I bring the entire list of items in the powerbi table?

[sag129]

Hi sekar, i just tried a page with 'load more' and it works as it is. can you share an example of the output you are getting? the 'load more' does not restrict like 'next page' which typically has a different URL to extract the rest of the rows from the table. Hence, when just 'load more', powerbi should be able to scrap the whole table even if it needs multiple clicks of 'load more' when you review the webpage manually. But you mentioned, you are trying to scrap amazon.com which I believe has an infinity scroll. Behavior needs to be evaluated but can do and share feedback here if works.

Thanks

Sagnik

https://linkedin.com/in/sagnikdatta 

[v-easonf-msft]

Hi  , @Anonymous 

Do you have resolved it? If yes, you could accept the helpful answer as solution. You also could share your own solution here. For now, there is no content of description in the thread. If you still need help, please share more details to us.

Best Regards,
Community Support Team _ Eason

[amitchandak]

@Anonymous ,refer if this can help

https://docs.microsoft.com/en-us/power-bi/connect-data/desktop-connect-to-web-by-example

https://blog.pragmaticworks.com/web-scraping-with-power-bi-python

[Anonymous]

I understand how to do web scrapping but my issue is I want to load the entire content. as I said in am trying to scrap a retail website page and for example laptops and the site shows only 10 items but when we click to show more it shows more items, but in powerbi table am seeing only the initial 10 items, not the whole list.

[Anonymous]

Hello, did you succeed ? 

[v-easonf-msft]

Hi, @Anonymous 

It doesn't support it currenctly. You only can extract the data  which appears in initial page .

It is suggest to  copy the data to a local Excel sheet, then import the Excel to PowerBI.

You also can check if it provides any Rest API to get the data.

Best Regards,
Community Support Team _ Eason