Forum Discussion
Using 2 functions to web scrape
I am scraping cost to own data from edmunds. I used the site map to get all the Cost to own links site map here. so I have a table of links and I used a function to get the model names from each of ...
Your URL is incomplete. You need to add the style query parameter
2016 Lexus LS 460: True Cost to Own | Edmunds
https://www.edmunds.com/lexus/ls-460/2016/cost-to-own/?style=401580679
Here's a potential way to get that value:
let
Source = Xml.Tables(Web.Contents("https://www.edmunds.com/sitemap_web54-mmy-cost-to-own.xml")),
Table0 = Source{0}[Table],
#"Kept First Rows" = Table.FirstN(Table0,10),
#"Added Custom" = Table.AddColumn(#"Kept First Rows", "Custom", each Web.BrowserContents([loc])),
#"Added Custom3" = Table.AddColumn(#"Added Custom", "Custom.3", each try Text.Range([Custom],Text.PositionOf([Custom],"<optgroup"),Text.PositionOf([Custom],"</optgroup>")-Text.PositionOf([Custom],"<optgroup")+11) otherwise "<optgroup/>"),
#"Parsed XML" = Table.TransformColumns(#"Added Custom3",{{"Custom.3", Xml.Tables}}),
#"Expanded Custom.3" = Table.ExpandTableColumn(#"Parsed XML", "Custom.3", {"option"}, {"option"}),
#"Expanded option" = Table.ExpandTableColumn(#"Expanded Custom.3", "option", {"Element:Text", "Attribute:value"}, {"Model", "Style"})
in
#"Expanded option"
Then you can fetch the true URL for each model variation to get the cost table.
Thank you! I posted my function code to get the model name. Clearly your code is more efficient. But I was confused about how to have pull the link and then the model to get the specific cost to own. I tried to calify in my orogonal question.
As I said you can use my code as the basis to get the model numbers. Then you can use your approach to fetch the table for each of the models.