Forum Discussion

ToddChitt's avatar
ToddChitt
Super User
2 years ago
Solved

Notebook fails with error 200 when run from Pipeline

I'm fairly new to Spark Notebooks. I have one that dumps a JSON file into a couple of tables in my Lakehouse. It works fine when the Notebook is run on its own. But when I try to run it from the cont...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi ToddChitt 

    Thanks for the details. Can you please send me the code which you are running in the notebook? I will try to repro this from my side and let you know.

    And also can you please try to create a new notebook and then attach the lakehouse. Now try to run this notebook from the pipeline. If the issue still persists please do let me know. This way there wont be a grey out option and you can attach or detach the lakehouse.


    Did you try to set the target lakehouse as the default lakehouse? Default lakehouse is identified by the pin icon.

     



    Thanks.

ToddChitt's avatar
ToddChitt
Super User
2 years ago

Hello Anonymous and thank you for the prompt reply.

Per your suggestion I created a new stand-alone notebook (i.e.: NOT from already inside the lakehouse) then manually ADDED the lakehouse as a new source. 

 

This worked! Invoking the Notebook from the context of a Pipeline run just fine.

 

Thank you. 

Do you still want the code? All it does is read a JSON file, parse it into 4 data frames and save each to a table in the lakehouse.

 

 

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi ToddChitt 

Thanks for the code. Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

  • ToddChitt's avatar
    ToddChitt
    Super User
    2 years ago

    I would consider this a bug: A Notebook created from the context of a Lakehouse has trouble access that lakehouse when run from inside a Pipeline. But a Notebook created on its own, with the Lakehouse added afterwards has no issues. 

     

    Please don't tell me that this was "by design"!

     

    • dpombal's avatar
      dpombal
      Post Partisan
      2 years ago

      This is clearly a bug, I managed to make it work creating a new empty notebook , but this should be fixed

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        I also experienced this issue and agree it is a bug. However an alternative workaround to creating a new notebook is to completely remove and add the linked lakehouse back. To do this you can select the default pinned lakehouse and select Remove all Lakehouses. Once added back it seems to trigger from pipeline with no errors.

         

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    hello v-nikhilan-msft,

    I am getting similar error but it is about variable/dataframe note defined.. Name Error. When I ran notebook manually, it is successfull but when triggered from pipeline it is failing with exceptions mentioned above.

     

    below is the line of code causing error, where read_particular_day_files is a user defined function.

    numfiles_pti_ar, df_toprocess_pti_ar, df_policy_transaction_info_ar = read_particular_day_files(accountingDate, '', 'PolicyTransactionInfo')
     
    please help me resolve this issue
    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi Anonymous 

      Thanks for the post. But as the initial ask is different from your issue, can you please create a new post and tag me. I will surely help.

      Please attach some screenshots of the error too.

      Thanks.