Forum Discussion

AmarRK2512's avatar
AmarRK2512
Frequent Visitor
1 year ago

Error Running Notebook in Pipeline

Hello, i'm quite new with data pipeline, my pipeline basically it's just insert and delete data looped for some range of dates, from redshift to lakehouse. So before insert data it will delete data for the same date in the lakehouse, but i got error in delete data process in some dates but no error in other dates between range looping. I already try running it manually for specific date that shows error and it work just fine. Any idea what is the error about? Currently it running daily with range date 45 days before today, and since there is some date with error in delete process, i need manually running for those dates to make it up to date.

 

Here is the error shows in delete process notebook  :
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o324.sql.
: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.apache.spark.sql.delta.actions.CommitInfo["timestamp"])

and here is the error message inside notebook :

System cancelled the Spark session due to statement execution failure

When i check the details running inside notebooks, all jobs status shows succeded, but the notebook running status shows failed.
Here i attach my screenshot data pipeline flow.

 

Activities inside, where "date to loop" as input parameter to notebook delete data

 

 

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi,
    Can you please provide more information with screenshots? would be helpful to understand and debug!

    • AmarRK2512's avatar
      AmarRK2512
      Frequent Visitor

      Hi, i already edit the post and attach few screenshot, hope it would be helpful!

      • Anonymous's avatar
        Anonymous
        Not applicable

        Hi AmarRK2512,

        In fact, I can't report your scenario based on the screenshot. Can you please share some detail scripts that you processing in these steps?

        BTW, the error message metions the delete data operation failed, I think you can add some operations to check if these data existed at first.

        Regards,

        Xiaoxin Sheng