Forum Discussion

kirah2128's avatar
kirah2128
Resolver I
2 years ago
Solved

One Large Text File Parsing 200MB Error 500

Dear Community, Have you encountered this problem? I'm processing a text file and using the code below. The script is running successfuly on my laptop but with Fabric it's not working. Do you have o...
data engineering
data warehouse
  • kirah2128's avatar
    kirah2128
    2 years ago

    Dear Everyone,

     

    the problem is solved =D by removing the print() function. 

    # print(contents)

    For some reason the fabric can't handle large concatinated string. 

HimanshuS-msft's avatar
HimanshuS-msft
Microsoft Employee
2 years ago

Hi kirah2128 

Since small files are working fine and only the big files are failing , I suggest you to work with the Microsoft support team on this . The reason being that they will have the details as what is the real error under the hood . At this time I do not think that cluster resource can be an issue as the spark is deisgned to handle big loads , Once you have created a support ticket , please do share the same here so that we can also keep an eye on the same .

Thanks 
Himanshu 

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi kirah2128 ,

    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Did you got a chance to create a support ticket? If you have created a support ticket , please do share the same here so that we can also keep an eye on the same.