Power BI Incremental Refresh with Python Script Data Source

[Leroy]

Hello everyone!

I'm new to Power BI and I need some advice: at the company where I work, we deal with a large volume of data. To create a dataset for analysis in Power BI, we use a Python script that fetches data from SQL databases and transforms it. The data needs to be updated daily, which takes a lot of time, and I'm thinking about how to speed up the report refresh process.

I've been reading about Incremental Refresh and tried to implement the idea, but I couldn't get it to work. After adding parameters RangeStart and RangeEnd to filter dates in the required table, for some reason, the refresh won't happen because I supposedly didn't add the parameters (even though they are added).

Does anyone have any ideas on how to speed up report refreshes? Has anyone managed to configure Incremental Refresh for data sourced from a Python Script?

I would be very grateful for any advice) 

[v-yilong-msft]

Hi  @Leroy ,

Have you solved your problem? If so, can you share your solution here and mark the correct answer as a standard answer to help other members find it faster? Thank you very much for your kind cooperation!

Best Regards

Yilong Zhou

[lbendlin]

You can create an Incremental Refresh setup against the SQL database.  It is not very productive to try this against the Python script as the query will not fold and there will be no performance benefit.

[Leroy]

Hi @lbendlin 
We cannot directly connect to SQL databases in Power BI. A Python script collects data from our databases into one dataframe for analysis, which we then place into Power BI for visualization.

As I understand it, if the data source is a Python script, there is no way to configure Incremental Refresh in Power BI at all, right?

[lbendlin]

That's not what I said.  You can still configure incremental Refresh but it won't give you any performance improvement.