topic Re: Notebook execution failed at Notebook service with http status code - '200' in Data Engineering

Notebook execution failed at Notebook service with http status code - '200'

aravindhsam8 — Tue, 22 Oct 2024 13:56:21 GMT

I have a Master orchestration pipeline, which consist of 6 invoked pipelines running parallelly.

Within each invoked pipeline, there are 3 notebooks running in a series (ingestion notebook--->Silver Notebook ---> Gold Notebook).

I'm facing the below issue often in the random silver or gold notes books sparksql statement.

FYI,The tables provided in the statements are not used in any other notebooks, each notebook has it's own set of table list.

This is not only specific to "create table statement", it also occurs for "drop table if exist" statement too.

Master orchestration runs everyday which is schedule to run daily and this issue persist at least weekly thrice.

Thanks in advance.

Re: Notebook execution failed at Notebook service with http status code - '200'

lbendlin — Tue, 22 Oct 2024 22:04:45 GMT

This is not only specific to "create table statement", it also occurs for "drop table if exist" statement too.

I think these issues are related. Looks like your code sometimes fails to drop the table, and then it complans that it cannot create an existing table.

Re: Notebook execution failed at Notebook service with http status code - '200'

Anonymous — Wed, 23 Oct 2024 05:41:27 GMT

Hi @aravindhsam8 ,

Thanks for the reply from lbendlin .

Can you check the data pipeline and the owner of the notebook? A similar problem may occur when the owner of the notebook is removed from the workspace. You can also check if the owner of the pipeline and notebook has access to all the items you need for your run.

Make sure you are using the Spark 1.3 runtime.

Can you try to add a retry run pipeline? See if the problem can be reproduced.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Notebook execution failed at Notebook service with http status code - '200'

aravindhsam8 — Wed, 23 Oct 2024 12:53:56 GMT

Hi @Anonymous ,
Thanks for the reply.

I have changed to Spark 1.3 runtime. I have also checked all the ownership of the items within the pipleine, everything are under my name.

I have tried to provide retry option if failed for each notebook activity, the issue still persist, it fails on the first and succeed either on second or third.
But still the notbook fails, even though all the above options are checked.

Re: Notebook execution failed at Notebook service with http status code - '200'

aravindhsam8 — Mon, 28 Oct 2024 11:12:04 GMT

HI @Anonymous , Just following back on this.
Kindly help on the above issue.
Thanks in advance.

Re: Notebook execution failed at Notebook service with http status code - '200'

aravindhsam8 — Mon, 04 Nov 2024 09:23:55 GMT

HI @Anonymous , Just following back again on this.
Kindly help on the above issue.
Thanks in advance.

Re: Notebook execution failed at Notebook service with http status code - '200'

richbenmintz — Mon, 04 Nov 2024 18:02:32 GMT

Are you able to slighlty refactor your code and use try -> except blocks in pyspark notebook with your sql code running in spark.sql(""), you can then catch the errors and see exactly what is going on.

You might also want to use consider using pyspark to load your bronze data into dataframe and write to dest lakehouse.

df = spark.read.format('delta').load('path_to_source_table') df.write().format('delta').mode('overwrite').saveAsTable('path_to dest_table')