Faster scraping data from eshop

[Petanek333]

Hi, 

I created a query to get data from czech eshop Alza.cz (like a czech version of Amazon). This is the way I did it:

1) I have a table with various columns, two of them being Product name and a direct link to that product.

2) I created (via Get data from web) a query to extract price with VAT (basicaly the only information I need) of a product. I used a feature Add Table Using examples to get price with VAT of this product: Product . Then using Advanced Editor I created a function from this query, pretty standard so far:

(example_source_link)=>
let
    Source = Web.BrowserContents(example_source_link),
    #"Extracted Table From Html" = Html.Table(Source, {{"Column1", ".bigPrice"}})
in
    #"Extracted Table From Html"

3) Then, using Add Column > Invoke Custom Function I added a column with this function having the column with the link to that product as an argument. Again, this is still pretty standard. 

Problems: 

1) The ehop uses several elements to display the price in various products. ".bigprice" mentioned above is only one of them (this is the element of price with discount), I found 6 different elements so far. So i created 6 columns with Invoke function and each line has to load the link 6 times if I am not mistaken about how it works.

After that I use DAX to select the ideal column containing a price to display it, but this part is not a subject of my question.

2) I have about 600 products (=600 links) and I am planning to add other eshops as well. Even with 200 products the query takes about 20 minutes to load.

Is there any other way or a best practice to get the prices from products? I cannot make one page of all the products as a view in this eshop, so I have to use direct links to those products. But I would like to know how to make it faster even using completely different way to get those elements (prices with VAT) to my Power BI Desktop 🙂

[lbendlin]

There is no faster way in Power BI. Ask the owners of the web shop if they offer an API.