Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
7 years ago
Solved

Website scraping advice

I'm trying to scrape a website with a pretty simple HTML table, but it uses Javascript for pagination and I can only get the first 25 results when using the web connector. I've tried using 

 [WaitFor = [Timeout = #duration(0,0,0,0)]])

to see if Power BI could pick up the table before the javascript loads -- I'm not sure if that's how it works but it hasn't given me any results yet.

 

Is there anything I can do? This is the website and data in question: 

http://www.onequestionshootout.xyz/episodes/series_all.htm

Need Help
Tutorial Requests
  • v-yuta-msft's avatar
    v-yuta-msft
    7 years ago

    Anonymous ,

     

    I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:

    https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts

     

    Community Support Team _ Jimmy Tao

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

6 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    7 years ago

    I'm afraid my skills at this point won't allow for python scripting, so in the meanwhile I've downloaded the page as .html and I used the Text/CSV data connector to get the table in plain HTML. The downside of course is I cannot get the latest updates to my report over the internet.

    • PaulDBrown's avatar
      PaulDBrown
      Community Champion
      6 years ago

      Anonymous 

      Wow! Thank you for sharing the idea of dowloading the page as an HTML! I was having the same problem as you and was completely stuck. With your solution, I have at least succeeded in extracting a "snapshot" of the data as it stands currently, which is better than no data at all...

      I would have never thought of downloading the actual page!

       

      Thanks!!

    • v-yuta-msft's avatar
      v-yuta-msft
      Community Support
      7 years ago

      Anonymous ,

       

      Power query only support simple web scrapying. If the website needs dynamic scrapying, I'm afraid power query won't work.

       

      Community Support Team _ Jimmy Tao

      If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

    • Anonymous's avatar
      Anonymous
      Not applicable
      7 years ago
      It started as promising, but unfortunately I can't get any parameters from the url as it doesn't produce any when you navigate through the pages... Tricky!
      • v-yuta-msft's avatar
        v-yuta-msft
        Community Support
        7 years ago

        Anonymous ,

         

        I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:

        https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts

         

        Community Support Team _ Jimmy Tao

        If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.