Forum Discussion
Query Experts needed - Iterative web scraper running slow
- 3 years ago
This code takes about 6 seconds per 100 entries. So about half an hour to fetch everything. Adjust the page range as needed.
let Source = {0..20}, Extract = (page)=> Html.Table(Web.BrowserContents("https://www.metacritic.com/browse/games/score/metascore/all/all/all?view=condensed&sort=desc&page=" & Text.From(page)), {{"Rank", ".numbered"}, {"Name", ".numbered + *"}, {"Platform", ".next_to_side_col .data"}, {"Released", ".platform + *"}, {"User Score", ".user"}, {"Critic Review", ".score:nth-child(1)"}, {"Summary", ".summary"}}, [RowSelector="TR"]), #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Added Custom" = Table.AddColumn(#"Converted to Table", "Custom", each Extract([Column1])), #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Rank", "Name", "Platform", "Released", "User Score", "Critic Review", "Summary"}) in #"Expanded Custom"How to use this code: Create a new Blank Query. Click on "Advanced Editor". Replace the code in the window with the code provided here. Click "Done".
- Anonymous3 years ago
That worls wonders. Thank you!
Would it be possible for it to also determine how many pages are actually present, right now I think it is 271, and then make that number dynamic based on this? Thanks again.
That worls wonders. Thank you!
Would it be possible for it to also determine how many pages are actually present, right now I think it is 271, and then make that number dynamic based on this? Thanks again.
- lbendlin3 years agoSuper User
Use List.Generate with an exit condition when the page doesn't return any more payload data. The disadvantage of that approach will be that you have to lug the entire resultset around inside the List.Generate, so you better have plenty of memory. Or you could play dirty tricks and keep trying to fetch pages, throw them away, determine the last page, then run the original code in the hope that your web browser cache has all the data locally available.