AUTO Refresh Power BI report that contains Web Scraping Function

[datacauts]

hi.

I'm doing a Power BI report about Brazilian economy and it's stocks market, IBOVESPA.

I managed to get all the data I'm using from excel queries and google sheets (google finance and importhtml), and there was a need for showing a newsletter about every company on IBOVESPA in the report.

After doing all the rest, I got this task and created a function that searches the news on Google News for a company and transform it into a table (with Journal, News Title, News Info, Date and URL)

Then, I created a table with all the companies and generated a single link for every one using M (like" http google search code" & [Company Name] & "elseelseelse dot com") and applied the function, generating the structured data I needed. Now, every time I refresh my report, the BI goes and re-do the function, bringing only the latest 10 news about every single company (almost 4600 results)

So the task was complete but then I need to put this report online and Power BI Service doesn't allow me to create a auto-refresh (scheduled), because it says it has dynamic functions (or, when I open the Datasource configuration in BI Desktop, it says that I have created hand-written queries).  

I tried searching for root of the problem and discovered that this function that web scraps the news is the one causing the problem (when I remove it, and, consequently, the entire Newsletter table, the Service allows me to schedule refresh).  

Is there a way to solve this, allowing the Newsletter to be auto update just like the rest of the data? Like, maybe, using the function to save the information on a new table that not depends on it or web scraping the News data via Google Sheets or something like that. I don't know, I'm really lost in this specific question.  

[Anonymous]

@datacauts did you found your solution? I am kind of facing the similar issue. 

[datacauts]

I didn't tried it because I abandoned the project... but I saw that for API cases you use the RelativePath in the initial calling function and then it allows to auto refresh. Search about it.

https://medium.datadriveninvestor.com/setting-a-scheduled-refresh-on-a-dynamic-data-source-in-power-...

[Anonymous · Accepted solution]

Hi  @datacauts  ,

A dynamic data source is a data source in which some or all of the information required to connect cannot be determined until Power Query runs its query, because the data is generated in code or returned from another data source. Examples include: the instance name and database of a SQL Server database; the path of a CSV file; or the URL of a web service.

In most cases, Power BI datasets that use dynamic data sources cannot be refreshed in the Power BI service. There are a few exceptions in which dynamic data sources can be refreshed in the Power BI service, such as when using the RelativePath and Query options with the Web.Contents M function. Queries that reference Power Query parameters can also be refreshed.

For detailed information about the content, you can check this link:

https://docs.microsoft.com/en-us/power-bi/connect-data/refresh-data#refresh-and-dynamic-data-sources

Best Regards,

Liu Yang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[datacauts]

🤔