Notebook fails with error 200 when run from Pipeline but manual run is successful

[pshepunde]

Every time, it is failing with error like variable/dataframe not defined, when triggered from pipeline. But when I executed notebook manually, it ran without error. Please help me resolve this issue. Same pipeline ran successfully before.

Operation on target Bronze Layer failed: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o4944.throwExceptionIfHave.
: com.microsoft.spark.notebook.msutils.NotebookExecutionException: name 'numfiles_pti_aric' is not defined
---------------------------------------------------------------------------NameError  

[kirah2128]

I'm having the same issue. And I'm just running a very basic notebook to get data, transform and save to lakehouse. And when I tried to access the notebook and connect to session. I can't connect to any session. 

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook. LivySessionId: 822a6249-b536-43c7-9b4c-217086f86b7dNotebook: Notebook_8cc50b58-17e6-49a7-87a8-0146a0539674.' :

[pshepunde]

Hello @v-nikhilan-msft , please find the screenshot for more details. Thanks in advance

[pshepunde]

error message as below:

Operation on target Bronze Layer failed: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o4944.throwExceptionIfHave.
: com.microsoft.spark.notebook.msutils.NotebookExecutionException: name 'numfiles_pti_aric' is not defined
---------------------------------------------------------------------------NameError                                 Traceback (most recent call last)Cell In[37], line 2
      1 from notebookutils import mssparkutils
----> 2 if numfiles_pti_aric==0:
      3     print(f"No Files exists for PolicyTransactionInfoARIC for {accountingDate}")
      4     mssparkutils.notebook.exit("NoFileExists_ARIC")
NameError: name 'numfiles_pti_aric' is not definedYou can check driver log or snapshot for detailed error info!  

line of code as below: It is a function call which returns 2 dataframes and one numeric value

numfiles_pti_aric, df_toprocess_pti_aric, df_policy_transaction_info_aric = read_particular_day_files(accountingDate, '', 'PolicyTransactionInfo')

[v-nikhilan-msft · Accepted solution]

Hi @pshepunde 
Thanks for providing the details.
The error message you’re seeing is a NameError in Python, which means that the variable numfiles_pti_aric is not defined before it’s being used in the if statement. The line of code you’ve shown is calling a function read_particular_day_files which is supposed to return three values, one of which is numfiles_pti_aric. If this function call is executed before the if statement where numfiles_pti_aric is used, then this error should not occur.
Please ensure that the function read_particular_day_files is correctly defined and is indeed returning the expected values. Also, make sure that the function call is executed before the if statement.

Possible Solutions:

1. Check Function Call: Ensure that the function read_particular_day_files is called before the if statement and that it's assigning the return values correctly. The function should be returning three values, and you should unpack them into separate variables.
   
   
2. Define Variable Before Use: If numfiles_pti_aric is not supposed to come from the function call, you need to define it with an initial value before using it in the if statement. Just create the variable initially as numfiles_pti_aric=0 in the beginning of the code.
   
   

Hope this helps. Please let me know if you have any further questions.

[pshepunde]

Ideally, we don't have to declare varibales or dataframes before assignment, but in this case, I declared variables and dataframes and then use them in function call. It worked for now, Thanks a lot!

[v-nikhilan-msft]

Hi @pshepunde 
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks

[v-nikhilan-msft]

Hi @pshepunde 

Thanks for using Fabric Community.

Can you please provide the screenshot of the code along with the error message? This would help me to understand the issue better.

Thanks

[v-nikhilan-msft]

Thanks for the details @pshepunde 

Can you please send me the screenshot of the code used in the notebook?

Thanks