Forum Discussion

ToddChitt's avatar
ToddChitt
Super User
2 years ago
Solved

Notebook fails with error 200 when run from Pipeline

I'm fairly new to Spark Notebooks. I have one that dumps a JSON file into a couple of tables in my Lakehouse. It works fine when the Notebook is run on its own. But when I try to run it from the cont...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi ToddChitt 

    Thanks for the details. Can you please send me the code which you are running in the notebook? I will try to repro this from my side and let you know.

    And also can you please try to create a new notebook and then attach the lakehouse. Now try to run this notebook from the pipeline. If the issue still persists please do let me know. This way there wont be a grey out option and you can attach or detach the lakehouse.


    Did you try to set the target lakehouse as the default lakehouse? Default lakehouse is identified by the pin icon.

     



    Thanks.

ToddChitt's avatar
ToddChitt
Super User
2 years ago

Hello Anonymous 

The Notebook was created from within the Lakehouse where the tables are. It is already connected. As stated, it runs just fine by itself and populates the tables in the lakehouse. 

I cannot add it again via the method you describe above as it is already connected.

If I follow the steps for "+ Data Sources", I select "Lakehouses", then the radio buttong for "Existing lakehouse". The lakehouse that the notebook was created under is listed, but grayed out with the message: "This item is preselected and can't be unchecked."

For all intents and purposes, I have to assume that the Notebook is already joined/knows about/is connected to the lakehouse. 

 

The error mentions "Spark SQL queries" but none of my three code blocks uses Spark SQL, they all use the default of PySpark.

 

Regards,

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi ToddChitt 

Thanks for the details. Can you please send me the code which you are running in the notebook? I will try to repro this from my side and let you know.

And also can you please try to create a new notebook and then attach the lakehouse. Now try to run this notebook from the pipeline. If the issue still persists please do let me know. This way there wont be a grey out option and you can attach or detach the lakehouse.


Did you try to set the target lakehouse as the default lakehouse? Default lakehouse is identified by the pin icon.

 



Thanks.