Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago
Solved

Query Experts needed - Iterative web scraper running slow

Hi,   I have a small project working on a Power BI model for demoing features etc. The report used to run, but recently has become absurdly slow.   It first consists of this function:   let Me...
  • lbendlin's avatar
    3 years ago

    This code takes about 6 seconds per 100 entries.  So about half an hour to fetch everything. Adjust the page range as needed.

     

     

    let
        Source = {0..20},
        Extract = (page)=> Html.Table(Web.BrowserContents("https://www.metacritic.com/browse/games/score/metascore/all/all/all?view=condensed&sort=desc&page=" & Text.From(page)), 
           {{"Rank", ".numbered"}, {"Name", ".numbered + *"}, {"Platform", ".next_to_side_col .data"}, {"Released", ".platform + *"}, {"User Score", ".user"}, {"Critic Review", ".score:nth-child(1)"}, {"Summary", ".summary"}}, [RowSelector="TR"]),
        #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
        #"Added Custom" = Table.AddColumn(#"Converted to Table", "Custom", each Extract([Column1])),
        #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Rank", "Name", "Platform", "Released", "User Score", "Critic Review", "Summary"})
    in
        #"Expanded Custom"

     

    How to use this code: Create a new Blank Query. Click on "Advanced Editor". Replace the code in the window with the code provided here. Click "Done".

     

     

  • Anonymous's avatar
    Anonymous
    3 years ago

    That worls wonders. Thank you!

     

    Would it be possible for it to also determine how many pages are actually present, right now I think it is 271, and then make that number dynamic based on this? Thanks again.