Error Running Notebook in Pipeline

[AmarRK2512]

Hello, i'm quite new with data pipeline, my pipeline basically it's just insert and delete data looped for some range of dates, from redshift to lakehouse. So before insert data it will delete data for the same date in the lakehouse, but i got error in delete data process in some dates but no error in other dates between range looping. I already try running it manually for specific date that shows error and it work just fine. Any idea what is the error about? Currently it running daily with range date 45 days before today, and since there is some date with error in delete process, i need manually running for those dates to make it up to date.

Here is the error shows in delete process notebook  :
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o324.sql.
: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.apache.spark.sql.delta.actions.CommitInfo["timestamp"])

and here is the error message inside notebook :

System cancelled the Spark session due to statement execution failure

When i check the details running inside notebooks, all jobs status shows succeded, but the notebook running status shows failed.
Here i attach my screenshot data pipeline flow.

Activities inside, where "date to loop" as input parameter to notebook delete data

[Anonymous]

HI @AmarRK2512,

I'd like to suggest you take a look at following link about data pipeline limitations if they meet at your scenarios:

Data Factory limitations overview - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

[MonicaCVL]

Hi,
Can you please provide more information with screenshots? would be helpful to understand and debug!

[AmarRK2512]

Hi, i already edit the post and attach few screenshot, hope it would be helpful!

[Anonymous]

Hi @AmarRK2512,

In fact, I can't report your scenario based on the screenshot. Can you please share some detail scripts that you processing in these steps?

BTW, the error message metions the delete data operation failed, I think you can add some operations to check if these data existed at first.

Regards,

Xiaoxin Sheng