URL Scraping function to be limited to certain cells

[paulroses]

Hello everyone,

i have a table with 3 columns looking like this:

trackingurl | Order Number | Shipment Status

This table gathers the above columns from another "master" table that contains many more columns and several rows for each order number (one row per item in each order).

I created the function below so that it goes through every row looking for a URL, scrapes it and delivers the result to column "Shipment Status".

let GetResults=(trackingurl) =>

let
Origine = Web.Page(Web.Contents(trackingurl)),
Data0 = Origine{0}[Data],
Children = Data0{0}[Children],
Children1 = Children{1}[Children],
Children2 = Children1{1}[Children],
Children3 = Children2{0}[Children],
Children4 = Children3{0}[Children],
Children5 = Children4{2}[Children],
Children6 = Children5{0}[Children],
Children7 = Children6{0}[Children],
Children8 = Children7{0}[Children],
Children9 = Children8{0}[Children],
#"Rinominate colonne" = Table.RenameColumns(Children9,{{"Text", "Shipment Status"}})
in
#"Rinominate colonne"

in GetResults

What i wish to achieve, is for the function to only scrape the URLs that correspond to any "Shipment Status" different from "Delivered" in order to avoid going through every single row on every update.

Can anyone help with this?

Thank so much in advance

Paolo

[lbendlin · Accepted solution]

Power BI has no memory.  It will always start from scratch.  If you need to keep track of prior runs then you need to do that before the data hits Power BI.

[paulroses]

Yes, i know that it wouldn't be possible to modify PowerBI's logic, but i hoped to find a "thinking out of the box" solution. 

Maybe an extra table where "delivered" rows are moved to hence removed from the table where the function scrapes URLs?

[lbendlin]

I would think that "you need to do that before the data hits Power BI"  is pretty much the same as  "thinking out of the box" ?

[paulroses]

I guess 🙂

Thanks for your help!