I am trying to scrape data from a website that has tabulated data generated and presented based on a selection made on the website.
Let's use the Fortune 500 website as an example of the challenge I am attemptign to overcome.
When you go to the website for 2021:
https://fortune.com/fortune500/2021/search/
you are presented with the top 10 companies by default.
These 10 results can easily be scraped using Web.BrowserContents, but you will notice there is a section at the bottom of the page that defines the display span.
You can select from 5 - 100 companies to display.
I have no html experience, I am an excel guy, so excuse me if there is an obvious solution here, but I can't work out if there is a way to modify this in Power Query prior to generating the html snapshot that the table extraction runs from.
Looking at the developer tools in Chrome, i can see that the table is generated in a section called <div class="ReactTable"> which is dependedent on the span set in the following html section:
<span class="select-wrap -pageSizeOptions">
<select aria-label="rows per page">
<option value="5">5 rows</option>
<option value="10">10 rows</option>
<option value="20">20 rows</option>
<option value="25">25 rows</option>
<option value="50">50 rows</option>
<option value="100">100 rows</option>
</select>
</span>
I can't seem to select a different span though once i am in the Add Table Using Examples wizard (even though there seems to be a button in the selector).
Therefore, I am wondering, is it possible to use a modifier like [WaitFor =] in Web.BrowserContents to define the span when first connecting to the website?
