Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
4 years ago

How to use URLs extracted from a website as data source for another table in Power BI

I have a situation where I need to extract tables from 13 different links, which have the same structure, and then append them into only one table with all the data. This way, at first I extracted the links from a home page by copying the link from the respective hyperlink, and then import the data through the Web connector on Power BI. However, 3 months later, I realized that those links changed every quarter but the link from the homepage where they are listed remain the same.

 

This way, I did some research and I found out this video on YouTube (https://www.youtube.com/watch?v=oxglJL0VWOI), which explained how I can scrape the links from a website, by building a table with the header of the link as a column and the respective link as another column. This way, I can have the links automatically updated, whenever I refresh the data. The thing is that I am having issues to figure out how can I use this links to extract the data automatically without having to copy them one by one and then import the data using the Power BI Web connector (Web.BrowserContents). Does anyone can give me a hint of how can I implement this? 

P.S.: I tried query from the links table to use each relative link as a parameter but I was not able to do it.

 

Thanks in advance!

Need Help

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    4 years ago

    Hi Anonymous ,

    If the format of the URL has a certain pattern, please look at this.

    Solved: Dynamic connection to Web URL

     

    You considered using parameters before, which is the right direction, and it didn't work because dynamic M query parameters only support direct query sources, and web sources can only use import mode.

    Dynamic M query parameters in Power BI Desktop

     

    Best Regards,
    Gao
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly. If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

    How to get your questions answered quickly -- How to provide sample data

    • Anonymous's avatar
      Anonymous
      Not applicable
      4 years ago

      Hi Anonymous.

      Thank you for your quick reply, which was very insightful. 

      Unfortunately, the URLs don't have a specific pattern that I can parameterize.
      Moreover, I still have some doubts regarding the Web.BrowserContents() function. At this moment, I am inserting manually the link in the function: Web.BrowserContents("https://........."). However, I wanted to query a specific link, from my links table, for each output table.

      As I have a table with the desired links, as I mentioned above, I decided to convert the column with the links ("Links") to a list. This way, if I would do a Blank Query, and wrote = Links{0}, the output is exactly the first link in the list. However, if I try to use this on the function: Web.BrowserContents(Links{0}), the following error is shown:

      Thus, my question is how can I query a specific URL from a table, to use it as a web source to another table.

       

      Thanks in advance!