Forum Discussion

RockStarNomad's avatar
RockStarNomad
Frequent Visitor
6 years ago

Can't query all rows from website table where URL doesn't change with pagination

Hello, I'm trying to do a simple web query for a couple of sites, but however their tables are coded, it only queries the first page of the table rather than all results. When clicking to the next page of the table (or selecting to view all rows), the page URL doesn't change, which eliminates a lot of the solutions I've found online so far. The closest I got was using a program called Fiddler to look for hidden code changes and apply them to the URL, but I could never get Fiddler to actually read any URL changes when I clicked from page to page.

 

Here is one of the sites, pretty basic, nba.com. When I query this table, I only get 50 rows of results even though there should be over 400. Ideal solution would be to capture the URL/object in such a way as to simply show all results and query on that view, but that may not be possible. Any help is appreciated!

https://stats.nba.com/players/touches/?sort=TOUCHES&dir=1

 

Thanks!

Need Help

2 Replies

    • RockStarNomad's avatar
      RockStarNomad
      Frequent Visitor
      6 years ago

      Hello, thanks for the reply, but this solution only works when the URL changes with pagination (ie the URL contains a parameter like page number or date that changes as more results are shown). In the examples I'm looking to scrape, the URL is static (thus the pagination is controlled by an on-page application/code and not from the URLs themselves.