Web Scraping: Interacting with Website on Load

[jmillsjmills]

Hello,

So I am trying to pull in content from this website: 

https://www.guts.com/gb/sports/football-s1/spain-c934/la-liga-t5453/ud-levante-vs-fc-sevilla-e845107...

This link could expire as football match is tonight - but the layout is the same on any page for a match on the website.

My code is as follows:

By default, when the page loads, the section entitled "Popular" is expanded but the other sections are not):

Therefore my code successfully pulls in the content that can be seen in the section that is expanded by default:

However, I wish to be able to interact with this page before the web scrape takes place. Is this possible? I would like somehow to open other dropdowns. Say, for example, the "Corners and Cards" section. In the above screenshot, you can see there are no div classes available below the "accordion_contentinner" highlighted in grey.

When this section is clicked open, however, the code then looks like the below. Crucially, it has new div tags beneath the "accordion__contentinner" section where before this content was not visible.

As I am looking to scrape this content (which is initially hidden), my question is whether there is any means of automating this process whereby that dropdown is expanded, and the div content beneath can then be pulled in.

I am hoping something can be added in that "WaitFor" part of the text, although it would not be a case of just waiting for some element to load - I somehow need to trigger that specific interaction via the query process. Is there a way of setting the "hidden" parameter to "visible" or something along these lines?

Thanks so very much! Hopefully it is clear - but I will keenly answer any questions you may all have!

jmillsjmills

[v-eachen-msft]

Hi @jmillsjmills ,

Power BI could only get a static page. You could get request generated after click, it may be the URL you need. However, guts.com is unavailable in my territory. So I cannot do further research.