Data lakes loading millions of records loading. Fail

[Anonymous]

I have a problem. the data lakes gen1 the data is set as multiple countries. 

adl://_______________.azuredatalakestore.net/Collated/FSFA/AM1/US

adl://_______________.azuredatalakestore.net/Collated/FSFA/AP2/IN

adl://_______________.azuredatalakestore.net/Collated/FSFA/AM1/CA

countries like us have over a million records to load. 

if have to get all the America's data into one report i am trying to load US data, Canada data, Mexico data,...... and append.

The problem comes now when I apply or close and apply the load after running for hours (like reaching over 2 million records) 

"unable to read data from the transport connection: an established connection was aborted by the software in your host machine"

I run the load multiple times it gives me an error again and again. 

Any suggestions on how to resolve this issue.

[v-xicai]

Hi @Anonymous ,

From the error message, it shows something on your machine is blocking your connection. Try running it with firewall/antivirus disabled. This is not permanent, just for testing if this is setup issue or code issue. If it passes without the firewall/antivirus then you have to find a way to allow it through them - add it to the firewall exceptions, etc. If this doesn't work the next step is to take a network sniff to see if your request goes out and what's being send on the wire.

In addition, you may not publish a pbix file over 1GB to Power BI Service,  and in the case that your pbix file is less than 1GB when importing it to Power BI service, but the data model grows beyond 1 GB after a period of time, you will get a message when refreshing the dataset.

Best Regards,

Amy 

Community Support Team _ Amy

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

i think considering the situation, it is good to create tranformation outside power bi as i have to make changes to 20 to 30 Million records. From Azure Data Lakes to Power Bi directly could be adjusted with any other microsoft tool?

Please you have any suggestion let me know.

May be sql or other warehouse

[Anonymous]

The reply i got is in the language i can not ready, please help me with getting this response in english.