API Integration with Python Script

[PG_MTA_PowerBI]

Hello guys,

I'm trying to integrate my data directly from an API to PowerBI with a Python script (see below).

Im downloading two different sets of data with two similar scripts: One for the product details and one for the model details.

"""
# installation the source package

# pip install --extra-index-url Source@package.com

"""

import os
#=================
# LOGIN/SETTTINGS
#=================

os.environ["Source_API_ENDPOINT"] = "hidden"
os.environ["Source_API_USER"] = "hidden"
os.environ["Source_API_PASSWORD"] =  "hidden"
os.environ["Source_ENV"]= "hidden"

MAX_SIZE = 30
FETCH_ALL = True # False =  MAX_SIZE 


#=================
# imports
#=================

from Source.api.endpoints import ProductAPI, ProductModelAPI
import pandas as pd

gen_models  = ProductModelAPI.objects.stream(convert_to_model=False)


def fetch_data(gen_models):
    data = []
    
    if FETCH_ALL:
        data = [x for x in gen_models]

    else:
        data = []
        for rx in range(MAX_SIZE):
            data.append(next(gen_models))
    return data 
    
    
#=================
#convert to pandas dataframe
#=================

models = pd.DataFrame(fetch_data(gen_models))

For the integration of the product data there is no problem and all 12.000+ rows are loaded from the API into PowerBI.

Whereas the model data loads some time and then gives the following error:

I've already tried the script for the model data in powershell with an extra code-line to give me an csv-file and everything worked fine. I got my data set with all 7200+ models in a csv-file. So maybe the script might not be the problem here.

To further investigate the error in PowerBI, I limited the amount of models to download by varying the "MAX_SIZE" line and with which I was able to download up to 7200 models until the Error occured again.

I can't quiet get my head around why the script works for the bigger dataset of products but not for the smaller data set of models. And why I'm able to download parts of the model-set but not all of it without getting an Error.

I hope that some of you guys might have an idea regarding the topic.

Thanks in advance

Marcel

[amitchandak]

@PG_MTA_PowerBI,

print (models)

[PG_MTA_PowerBI]

Thanks @amitchandak for your answer, but this didn't fix the problem. Still same error.