Extract data from website

[J94]

Hi everyone,

I want to extract dividend data from the site dividendhistory. However, with underlying code, I am only able to extract a few rows instead of the whole table. And this stock has several pages without the website link is changing (see image below). How can the code be changed that it recognize everything? 

let Source =
Web.BrowserContents("https://dividendhistory.org/payout/KO/"),
ExtractedTable = Html.Table(Source, {{"Ex-Dividend Date", ".tabulator-cell[tabulator-field=""ex-dividend_date""]"}, {"Payout Date", ".tabulator-cell[tabulator-field=""payout_date""]"}, {"Cash Amount", ".tabulator-cell[tabulator-field=""cash_amount""]"}}, [RowSelector=".tabulator-row"]),
ChangeType = Table.TransformColumnTypes(ExtractedTable,{{"Ex-Dividend Date", type date}, {"Payout Date", type date}, {"Cash Amount", type text}}) in ChangeType

[danextian]

Hi @J94 

I have inspected the site using the developer tool. Navigating to a page or changing the number of items creates a  post request which real API endpoint is unfortunately hidden behind a javascript. 

If you change the page size to 100 there will only be 3 pages so extracting the data manually shouldn't take that long.

[PhilipTreacy · Accepted solution]

@J94 

That site is loading data with JavaScript so unfortunately you can't use Power Query to get the data.

PQ requires that the data can be retrieved with parameters that are supplied in the URL (or possibly in headers).  Unless the site API allows this you are stuck.

You could use somethign like Selenium to get the data but that requires further programming outside PQ.

There are webites that provide scraping as a service, you can search for them, but that will cost you and require integration into PQ.

Phil

[AliceW]

J, I had a similar problem and I used Grok to give me the precise answer. It worked.

Try it yourself.