Loading Data using python in Power BI throwing memory error

[Anonymous]

Hello,

I am trying to load data from multiple CSV files after appending them into one data frame into power bi using python, but I am getting a memory error. The same script, if I run through jupyter notebook, doesn't throw any memory error. If I try reducing the number of files then it works well in Power BI. I am not sure if this is due to any row limit in power bi or due to checkout time as the number of rows is up to 60 million and the time required to run the script is almost 1 hour in the notebook. 

The error is :

DataSource.Error: ADO.NET: Python script error.
Traceback (most recent call last):
File "PythonScriptWrapper.PY", line 129, in <module>
x_df = x_df.append(df, ignore_index=True)
File "C:\PROGRAMS\PYTHON\PYTHON37-32\lib\site-packages\pandas\core\frame.py", line 6692, in append
sort=sort)
File "C:\PROGRAMS\PYTHON\PYTHON37-32\lib\site-packages\pandas\core\reshape\concat.py", line 229, in concat
return op.get_result()
File "C:\PROGRAMS\PYTHON\PYTHON37-32\lib\site-packages\pandas\core\reshape\concat.py", line 426, in get_result
copy=self.copy)
File "C:\PROGRAMS\PYTHON\PYTHON37-32\lib\site-packages\pandas\core\internals\managers.py", line 2058, in concatenate_block_managers
[ju.block for ju in join_units], placement=placement)
File "C:\PROGRAMS\PYTHON\PYTHON37-32\lib\site-packages\pandas\core\internals\blocks.py", line 328, in concat_same_type
axis=self.ndim - 1)
MemoryError

Can someone please help me with this?

Thanks.

[v-eachen-msft]

Hi @Anonymous ,

Open Task Manager to check the memory usage and check if your memory is sufficient during operation.

[Anonymous]

@v-eachen-msft Yes I tried checking that and there is sufficient memory available to perform the task.  

[amitchandak]

Check the amount of memory allocation to Python.

[Anonymous]

@amitchandak  do you mean to say checking the amount of memory allocated to python on my system or in power bi?

[amitchandak]

I can see the error in python. So check for python. And I am assuming you are using python because you are not able to do in transform data/edit queries

[Anonymous]

@amitchandak  the error occurs only when running the python script in Power BI, I tried the same script in Jupyter notebook and its running fine without any error, just that it takes an hour to run the file. Is there any link between memory error and the time taken by the script to run? because the limitation of loading data using python/R scripts is that the execution time is limited up to 30 minutes.