Forum Discussion
Query Experts needed - Iterative web scraper running slow
- 3 years ago
This code takes about 6 seconds per 100 entries. So about half an hour to fetch everything. Adjust the page range as needed.
let Source = {0..20}, Extract = (page)=> Html.Table(Web.BrowserContents("https://www.metacritic.com/browse/games/score/metascore/all/all/all?view=condensed&sort=desc&page=" & Text.From(page)), {{"Rank", ".numbered"}, {"Name", ".numbered + *"}, {"Platform", ".next_to_side_col .data"}, {"Released", ".platform + *"}, {"User Score", ".user"}, {"Critic Review", ".score:nth-child(1)"}, {"Summary", ".summary"}}, [RowSelector="TR"]), #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Added Custom" = Table.AddColumn(#"Converted to Table", "Custom", each Extract([Column1])), #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Rank", "Name", "Platform", "Released", "User Score", "Critic Review", "Summary"}) in #"Expanded Custom"How to use this code: Create a new Blank Query. Click on "Advanced Editor". Replace the code in the window with the code provided here. Click "Done".
- Anonymous3 years ago
That worls wonders. Thank you!
Would it be possible for it to also determine how many pages are actually present, right now I think it is 271, and then make that number dynamic based on this? Thanks again.
This code takes about 6 seconds per 100 entries. So about half an hour to fetch everything. Adjust the page range as needed.
let
Source = {0..20},
Extract = (page)=> Html.Table(Web.BrowserContents("https://www.metacritic.com/browse/games/score/metascore/all/all/all?view=condensed&sort=desc&page=" & Text.From(page)),
{{"Rank", ".numbered"}, {"Name", ".numbered + *"}, {"Platform", ".next_to_side_col .data"}, {"Released", ".platform + *"}, {"User Score", ".user"}, {"Critic Review", ".score:nth-child(1)"}, {"Summary", ".summary"}}, [RowSelector="TR"]),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "Custom", each Extract([Column1])),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Rank", "Name", "Platform", "Released", "User Score", "Critic Review", "Summary"})
in
#"Expanded Custom"
How to use this code: Create a new Blank Query. Click on "Advanced Editor". Replace the code in the window with the code provided here. Click "Done".