Forum Discussion

kirah2128's avatar
kirah2128
Resolver I
2 years ago
Solved

One Large Text File Parsing 200MB Error 500

Dear Community, Have you encountered this problem? I'm processing a text file and using the code below. The script is running successfuly on my laptop but with Fabric it's not working. Do you have o...
data engineering
Data Warehouse
  • kirah2128's avatar
    kirah2128
    2 years ago

    Dear Everyone,

     

    the problem is solved =D by removing the print() function. 

    # print(contents)

    For some reason the fabric can't handle large concatinated string. 

kirah2128's avatar
kirah2128
Resolver I
2 years ago

To all, this is the latest discovery. They will go deeper invgestigation on this as we found this warning during the execution of the simple read and print of 200mb text file

 

2024-01-03 05:09:38,976 WARN ClientCnxn [Thread-68-SendThread(vm-fdb01374:XXXX)]: Session 0x10000008c0c0002 for sever vm-fdb01374/XX.X.XXX.X:XXX, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.

org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 73538ms for session id 0x10000008c0c0002

 

 

kirah2128's avatar
kirah2128
Resolver I
2 years ago

To all following this issue:, They asked us to increase the workspace setting, but before reaching them we already did and increase it to XXlarge node but still same issue.