Python crashes only after Transforming data + alternative for loading data when viewing mobile app?

[BeautifulDash]

I have Power BI run a Python script (that calls API endpoints) as a source. The Python script executes as expected and Power BI runs it well and loads all Pandas tables. Oddly enough, after I Transform the data (change type, rename) and even when I don't Transform but only click Apply & Close .. I get a Python error that indicates I likely exceed the API's max. rate for requests. How is it possible that after loading the data into Power BI and transforming it, it again executes the script to get the data AND runs into an unexpected issue. I didn't reach my max. number of API requests per day, and I cannot imagine that Power BI somehow executes a Python script/API calls faster than running the script on its own, plus the various functions have rate-limiters built-in. Any idea?

Also, loading the data takes a lot of time. I'd like to build a mobile report/an app that allows me to check my electricity consumption. Could we solve both issues by saving the data once and having it refresh regularly, prior to opening Desktop/Mobile?

[BeautifulDash]

Thank you. The script will load all historical data and then only load updates (eg every few minutes). What is the best way to store this data in the cloud, and have Power BI load from that cloud file, also to ensure that, when opening the mobile app, no API calls need to be made/the report loads fast? Do I store an Excel/CSV file in OneDrive, and connect Power BI? What is the standard way of doing this? 

[Anonymous]

Hi @BeautifulDash ,

Please try below steps:

1. API Call Optimization: Although you mentioned having rate limiters within your functions, it's worth reviewing the script to ensure that it's optimized for minimal API calls, especially if the script is executed multiple times for any reason. Techniques such as caching responses or using bulk API endpoints (if available) can reduce the number of calls.

2. Error Handling in Script: Implement robust error handling within your Python script to gracefully handle rate limit errors. This can include retry logic with exponential backoff or logging to help diagnose the issue further.

3. Improving Data Loading Times and Mobile Reporting
To address the issue of data loading times and building a mobile report for checking electricity consumption, consider the following:

4. Incremental Refresh: Utilize Power BI's Incremental Refresh policies to reduce the amount of data loaded with each refresh. This can significantly improve performance and reduce the load on your API.

5. Storing Data: Consider storing your data in a database or a cloud storage solution after the initial API call. This way, Power BI can connect to this data source instead of directly calling the API each time. This approach can also facilitate building a mobile report, as the data would be readily available for Power BI to consume.

Best regards,
Community Support Team_Binbin Yu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.