Web Scraping always bringing back first 10 line items only!

[yk1984]

Hello all,

I am trying to use the web scraping feature in Power BI to extract all open job positions from the following site: Estee Lauder Job Search - Jobs - Page 1 (elcompanies.com)

It shows the top 100 open vacancies but when ever i try to extract, it always only extracts the first 10 openings. I have also changed the page to show only 10 vacanices per page and then used a custom invoke logic to bring back the 10 openings per every page. When i do this it always shows me the same 10 job openings. Even if i enter the URL for page 2 of the vacancies, it still only brings back the first 10 vacancies from page one.

Has anyone faced anything like this before or know of a solution to get around this?

Your help would be appreciated, many thanks in advance

[DanBT]

Hey anychance you could explain how you solved this through a dataflow? I'm having the exact same problem with a different website. 

[yk1984]

What i did is use the web scraping as you would in desktop, copy the query and then paste into a dataflow blank query. This worked for me and showed all records, on desktop still shows on 10 records for some reason

[yk1984]

hello, yes i am on March 2020 version. Very strange that you get all 100 and i only keep getting 10 rows extracted. Are you just copying the web link using the Web data source?

Are you using the link https://jobs.elcompanies.com/estee-lauder-companies/jobs?page=1&limit=100

[Vijay_A_Verma]

Yes, same link. See the screenshot below to see more than 10 records

[yk1984]

ah thats so not fair its working for you but not me 😕

[v-yingjl]

Hi @yk1984 ,

It also works fine in my side with the same version of Power BI Desktop.

So it may relate to the client site of Power BI Desktop not the web source site.

Have you used the Microsoft store app version or the web download version of Power BI Desktop?

In addition, you can try to use the previous version of Power BI Desktop to check it again.

Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[yk1984 · Accepted solution]

Hello. I am using the version from the microsoft store. 

I have found a work around by bringing the data through data flow on the service which seems to have no issues in extracting all the data 😊

[Vijay_A_Verma]

I will suggest that you make a post to sister forum as well - Power BI Desktop Forum

[Vijay_A_Verma]

Not able to reproduce as I get all 100 through the above link. Are you on latest version of Power BI Desktop (Mar 2022)? See it in Help - About