Schedule a notebook to run automatically failed with the error code "InternalError"

[ZJoe]

Hello guys:

    I've developed a Fabric notebook for a specific mail sending task, and I've been scheduled it to run automaticlly(every Mon). but the last two run have failed, with a error code "InternalError" and no more logs.

    The notebook script should be fine, because I've clicked the "Run all" button after the schedule failed, and it worked fine. 

   Any pointers would be greatly appreciated.

[ZJoe · Accepted solution]

Hi @govindarajan_d  @v-pagayam-msft 

After a period of investigation, Microsoft support gave feedback yesterday that the issue was indeed the result of a bug. They said that my notebook name has some unsupported characters, such as Chinese characters, etc., and there is a parameter in their product that doesn't support those characters, so it will fail to run.

They will be working on a fix for this issue and the ETA is at the end of March. The current solution is to use only ASCII characters in the notebook name, in which case everything works fine!

[govindarajan_d]

Hi @ZJoe,

Thanks for the response. As assumed, it was a MS bug! 😆 And thanks for letting us know, this helps! 

[govindarajan_d]

Hi @ZJoe,

So the notebook works in interactive manner but not when you run it in a scheduled manner? 

Can you provide the script that you are using, without any sensitive information in it

[ZJoe]

Hi @govindarajan_d ,

Yes, the notebook works fine when running each cell individually or when click the "Run all" button to run all cells in sequence.

the script has too many lines and contains a lot of business logic, sorry for can't provide directly. but I can introduce the main components of the script for you.

the notebook is a "PySpark(Python)" language script, using a customized environment with some public libraries and custom libraries installed.

First, it uses Spark SQL to process some data and write the result into a lakehouse tabe, for subsequent use.

Second, it imports some libraries and retrieves data from the lakehouse table

Finally, using pandas to process data based on business logic, and wirte each result dataframe into a ".xlsx" file(in a loop), which saved in the Built-in resources folder. then using email library to send mail for each person, with the corresponding file as an attachment.

The above is the main content of the script, I'm wondering if the problem might have something to do with the environment or the amount of data?  

Thanks！

[govindarajan_d]

Hi @ZJoe,

Most probably it might have to do with environment rather than amount of data.

In your workspace settings, have you set the default environment to be the one that you use to execute interactively?

[ZJoe]

Thanks for your reply @govindarajan_d .

In my workspace, the Set default environment toggle is set to Off.  because not all notebooks in my workspace require the same libraries that are included in the environment I use for interactive execution, and it takes longer to start a new session for notebooks that don't need any library if set to On. so I just attach the environment to those specific notebooks.

In your opinion, the environment attached to notebook will be ineffective when running in a scheduled manner? 

[govindarajan_d]

Hi @ZJoe,

The environment selected should have been effective when running in a scheduled manner. That setting is not required, but I thought if the environment is used across all notebooks, we can try to enforce the environment being used just to see if that works. 

Two things: 

1. Can you select the default environment once and then change it back to test-environment and then see if that works? (Kind of like a reset)

2. In view all runs, you would be able to see the previous runs right. Can you go to the error runs

And see the log information here to see if you could find more information about the error?

[ZJoe]

Hello @govindarajan_d 

I make a copy of the notebook for test.

For first thing, I open the notebook, select the default environment, it shows a notification of "Update environment to Workspace Setting", then I change it back to my custom environment, also it shows a notification of "Update environment to py_env_1.3". after that, I set a schedule for it, and when reaching the specified time, the run is failed as before.

For second thing, it seems even stranger and I notice I missed it in the problem description. 

All the error runs by schedule only show in the Monitor panel, none of them shows in the View all runs, as if it's not running at all, see below. so again, except the information of "InternalError" shows in the Monitor panel, I can't find more helpful information..

[govindarajan_d]

Hi @ZJoe,

Ok. We can definitely say that the spark session itself did not start and the problem is in the environment. 

I would suggest creating a new environment with the same setting and see if that works ( this is all trial and error method at this point, since the obvious process doesn't seem to happen 😅 ).

Also, I would recommend creating a MS Ticket since this seems to be  a bug. 

[ZJoe]

Thanks for your suggestions @govindarajan_d 

I have created a MS ticket and will try to create a new same environment later. I'll update here as soon as there's new information from MS support.

Thanks again~

[v-pagayam-msft]

Hi @ZJoe ,

Has your issue been resolved after raising a support ticket? If so, kindly share the solution and mark it as accepted to assist other members in finding it more easily.

Thank you for being a part of Microsoft Fabric Community Forum!