topic Re: Error Running Notebook in Pipeline in Data Engineering

Error Running Notebook in Pipeline

AmarRK2512 — Wed, 06 Nov 2024 07:08:13 GMT

Hello, i'm quite new with data pipeline, my pipeline basically it's just insert and delete data looped for some range of dates, from redshift to lakehouse. So before insert data it will delete data for the same date in the lakehouse, but i got error in delete data process in some dates but no error in other dates between range looping. I already try running it manually for specific date that shows error and it work just fine. Any idea what is the error about? Currently it running daily with range date 45 days before today, and since there is some date with error in delete process, i need manually running for those dates to make it up to date.

Here is the error shows in delete process notebook :
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o324.sql.
: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.apache.spark.sql.delta.actions.CommitInfo["timestamp"])

and here is the error message inside notebook :

System cancelled the Spark session due to statement execution failure

When i check the details running inside notebooks, all jobs status shows succeded, but the notebook running status shows failed.
Here i attach my screenshot data pipeline flow.

Activities inside, where "date to loop" as input parameter to notebook delete data

Re: Error Running Notebook in Pipeline

Anonymous — Wed, 06 Nov 2024 06:04:34 GMT

Hi,
Can you please provide more information with screenshots? would be helpful to understand and debug!

Re: Error Running Notebook in Pipeline

Anonymous — Wed, 06 Nov 2024 06:57:37 GMT

HI @AmarRK2512,

I'd like to suggest you take a look at following link about data pipeline limitations if they meet at your scenarios:

Data Factory limitations overview - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

Re: Error Running Notebook in Pipeline

AmarRK2512 — Wed, 06 Nov 2024 07:09:24 GMT

Hi, i already edit the post and attach few screenshot, hope it would be helpful!

Re: Error Running Notebook in Pipeline

Anonymous — Fri, 15 Nov 2024 07:38:32 GMT

Hi @AmarRK2512,

In fact, I can't report your scenario based on the screenshot. Can you please share some detail scripts that you processing in these steps?

BTW, the error message metions the delete data operation failed, I think you can add some operations to check if these data existed at first.

Regards,

Xiaoxin Sheng