Website scraping advice

[Anonymous]

I'm trying to scrape a website with a pretty simple HTML table, but it uses Javascript for pagination and I can only get the first 25 results when using the web connector. I've tried using 

 [WaitFor = [Timeout = #duration(0,0,0,0)]])

to see if Power BI could pick up the table before the javascript loads -- I'm not sure if that's how it works but it hasn't given me any results yet.

Is there anything I can do? This is the website and data in question: 

http://www.onequestionshootout.xyz/episodes/series_all.htm

[Anonymous]

I'm afraid my skills at this point won't allow for python scripting, so in the meanwhile I've downloaded the page as .html and I used the Text/CSV data connector to get the table in plain HTML. The downside of course is I cannot get the latest updates to my report over the internet.

[PaulDBrown]

@Anonymous 

Wow! Thank you for sharing the idea of dowloading the page as an HTML! I was having the same problem as you and was completely stuck. With your solution, I have at least succeeded in extracting a "snapshot" of the data as it stands currently, which is better than no data at all...

I would have never thought of downloading the actual page!

Thanks!!

[v-yuta-msft]

@Anonymous ,

Power query only support simple web scrapying. If the website needs dynamic scrapying, I'm afraid power query won't work.

Community Support Team _ Jimmy Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[kcantor]

@Anonymous 

Perhaps this resource will help.

https://datachant.com/2017/03/30/web-scraping-power-bi-excel-power-query/

[Anonymous]

It started as promising, but unfortunately I can't get any parameters from the url as it doesn't produce any when you navigate through the pages... Tricky!

[v-yuta-msft · Accepted solution]

@Anonymous ,

I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:

https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts

Community Support Team _ Jimmy Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.