Forum Discussion
Web scraping - website changed
I think it might be an anti-spam thing on this forum but they shorten any URLs in posted stuff so we don't know what the 2 lines with web links are.
I had a brief try with the Query. The table that gets returned with fund names is results__table not sortableTable.
The second web call (RearragnedURL ) presumably constructs each url with positions in the list e.g. You can see that the &start= parameter controls which fund is displayed on the page
https://www.hl.co.uk/funds/fund-discounts,-prices--and--factsheets/search-results?sectorid=121&start=0&rpp=20&lo=0%2C1&sort=fd.full_description&sort_dir=asc
https://www.hl.co.uk/funds/fund-discounts,-prices--and--factsheets/search-results?sectorid=121&start=20&rpp=20&lo=0%2C1&sort=fd.full_description&sort_dir=asc
Overall, I think the work involved is too complex for a forum issue. It needs time and effort.
I appreciate you having a go HotChilli
Sadly, yes it is quite complex. I have spent most of the day manually scraping the website .... eeeek!
Yes, the table name has changed, apologies for not stating that in my original post. There are 39 "sectorid" and each sectorid, could possibly, have more than 1 page and with the change in the website you can't put page numbers in the URL. That's just another headache I have come across.
Anyway I appreciate you taking the time to look. I'll keep scratching my head and hopefully reach a solution
Thanks again
Another headache I faced was not able to navigate to the tab "Prices & Yields". This doesn't appear in the URL so I don't know how to access this tab via the query.
Many thanks
