Forum Discussion

danextian's avatar
danextian
Icon for Super User rankSuper User
9 years ago
Solved

How to scrape image URL from a webpage?

Hi All,   Is there a way to get the image URL from a webpage? From example, the poster images from IMDB. I don't want to manually right click each of them and copy the image url.
  • Anonymous's avatar
    Anonymous
    9 years ago

    Just for fun, I scraped together some pretty abhorrent M using the GUI, pointed at a wikipedia article. Here's what it looks like if you scrape all images from https://en.wikipedia.org/wiki/The_Lord_of_the_Rings:_The_Return_of_the_King

     

     

    After all the mangling and filtering, the M came out like this:

    let
        Source = Table.FromColumns({Lines.FromBinary(Web.Contents("https://en.wikipedia.org/wiki/The_Lord_of_the_Rings:_The_Return_of_the_King"), null, null, 65001)}),
        #"Filtered Rows" = Table.SelectRows(Source, each Text.Contains([Column1], "src=""//upload")),
        #"Split Column by Delimiter" = Table.SplitColumn(#"Filtered Rows", "Column1", Splitter.SplitTextByEachDelimiter({"src=""//"}, QuoteStyle.None, true), {"Column1.1", "Column1.2"}),
        #"Changed Type" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Column1.1", type text}, {"Column1.2", type text}}),
        #"Split Column by Delimiter1" = Table.SplitColumn(#"Changed Type", "Column1.2", Splitter.SplitTextByEachDelimiter({""""}, QuoteStyle.None, false), {"Column1.2.1", "Column1.2.2"}),
        #"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter1",{{"Column1.2.1", type text}, {"Column1.2.2", type text}}),
        #"Removed Columns" = Table.RemoveColumns(#"Changed Type1",{"Column1.1", "Column1.2.2"}),
        #"Filtered Rows1" = Table.SelectRows(#"Removed Columns", each Text.EndsWith([Column1.2.1], ".jpg") or Text.EndsWith([Column1.2.1], ".png") or Text.EndsWith([Column1.2.1], ".gif")),
        #"Added Custom" = Table.AddColumn(#"Filtered Rows1", "https", each "https://"),
        #"Reordered Columns" = Table.ReorderColumns(#"Added Custom",{"https", "Column1.2.1"}),
        #"Merged Columns" = Table.CombineColumns(#"Reordered Columns",{"https", "Column1.2.1"},Combiner.CombineTextByDelimiter("", QuoteStyle.None),"Merged"),
        #"Renamed Columns" = Table.RenameColumns(#"Merged Columns",{{"Merged", "Images"}}),
        #"Duplicated Column" = Table.DuplicateColumn(#"Renamed Columns", "Images", "Images - Copy"),
        #"Renamed Columns1" = Table.RenameColumns(#"Duplicated Column",{{"Images - Copy", "ImageURLs"}})
    in
        #"Renamed Columns1"

    There's definitely a few things I could do here to clean it up, but the point is, you're going to have to load the webpage as a text file and start filtering down to context around the link you need. Once you isolate the links as full URLs, you can set the column type to "Image URL" and use those images in your Power BI report.