Web Scraping table that is loaded by the web site

[Anonymous]

Hi!

I am trying to access some data through web scraping. The data are shown here: https://www.udir.no/tall-og-forskning/statistikk/statistikk-grunnskole/resultater-fra-elevundersokel...

What I have tried to do is to import data with the "Web"-function in Power BI. However, that only returns the main menu as a table.

I was quite confused at first, but after digging around in the source code, it seems that the table is loaded in through some javascript and is not present in the source code as a table. However, once the table is generated and I inspect that element, I can find a HTML-table.

After searching the web for answers, I tried "Add table using example", but this didn't work.

I have tried searching for other ways to do this, but I can't seem to find an answer.

Does anybody know how I might extract this table?

Thank you in advance!

[PhilipTreacy]

Hi @Anonymous 

What I did was the Export button

Then save the CSV.  Then in your browser ifd you look at your list of Downloads, you can right click the file and then copy the download link

I'm using Edge, I also tested in Chrome and it allows you to do the same.

Regards

Phil

[PhilipTreacy · Accepted solution]

Hi @Anonymous 

If the tables are loaded into the web page with JavaScript, unfortunately PBI can't see them.

You could try downloading the CSV file from that page which you can get by clicking on the Export button.

This is the CSV file URL

https://statistikkportalen.udir.no/api/rapportering/rest/v1/Statistikk/GSK/ElevundersoekelsenG/1/3/data.csv?radSti=**&filtre=EierformID(-10)_EnhetID(-12)_IndikatorID(-63_-34_-29_-27_-25_-23_-21_-19_-17_-15_-13)_KjoennID(-10)_TidID(201901_202001_202101)_TrinnID(6_9)_VisStandardavvik(0)_VisSvarprosent(0)&dataChanged=2021-09-30_131752

and this is the query to get it

let
    Source = Csv.Document(Web.Contents("https://statistikkportalen.udir.no/api/rapportering/rest/v1/Statistikk/GSK/ElevundersoekelsenG/1/3/data.csv?radSti=**&filtre=EierformID(-10)_EnhetID(-12)_IndikatorID(-63_-34_-29_-27_-25_-23_-21_-19_-17_-15_-13)_KjoennID(-10)_TidID(201901_202001_202101)_TrinnID(6_9)_VisStandardavvik(0)_VisSvarprosent(0)&dataChanged=2021-09-30_131752"),[Delimiter="	", Columns=11, Encoding=1200, QuoteStyle=QuoteStyle.None]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}, {"Column5", type text}, {"Column6", type text}, {"Column7", type text}, {"Column8", type text}, {"Column9", type text}, {"Column10", type text}, {"Column11", type text}})
in
    #"Changed Type"

Regards

Phil

[Anonymous]

Thank you, Phil! I am trying to retrace your steps, but I can't seem to find the URL for the CSV. Could you please tell me how you found it? I'll be sure to update your solution as the solution as soon as I have tested it.