PQ scrap data

Anonymous · ‎03-14-2021

Hi!

I want to scrap data from web page with ratings and reviews of the clients. However, there is a button "Show more", which doesn't change URL link, but expand number of reviews.

Power Query capable of scraping from what is shown on the default web page instance. So, even if you were to click and expand the data set to 100 rows, the Power Query engine is still only going to detect the first 10.

What can I do to fix this problem?

PhilipTreacy · ‎03-15-2021

Hi @Anonymous

This Show more button may not actually be loading more data, it may just be displaying it on screen. So the data you are after may be in the page, just not displayed. I can't say for sure without knowing the URL and checking it.

But if the Show more button does load more data then you won't be able to use PQ to interact with the page and click that button.

To do this you will need to use something like VBA and/or Selenium to drive a web browser and get your data.

Further info on that approach here

Web Scraping - Filling in Forms • My Online Training Hub

regards

Phil

Did I answer your question? Then please mark my post as the solution.
If I helped you, click on the Thumbs Up to give Kudos.

Blog :: YouTube Channel :: Connect on Linkedin

Proud to be a Super User!