Forum Discussion
Notebook execution failed at Notebook service with http status code - '200'
I have a Master orchestration pipeline, which consist of 6 invoked pipelines running parallelly. Within each invoked pipeline, there are 3 notebooks running in a series (ingestion notebook--->Silver...
Are you able to slighlty refactor your code and use try -> except blocks in pyspark notebook with your sql code running in spark.sql(""), you can then catch the errors and see exactly what is going on.
You might also want to use consider using pyspark to load your bronze data into dataframe and write to dest lakehouse.
df = spark.read.format('delta').load('path_to_source_table')
df.write().format('delta').mode('overwrite').saveAsTable('path_to dest_table')