Forum Discussion
Schedule a notebook to run automatically failed with the error code "InternalError"
Hi govindarajan_d Anonymous
After a period of investigation, Microsoft support gave feedback yesterday that the issue was indeed the result of a bug. They said that my notebook name has some unsupported characters, such as Chinese characters, etc., and there is a parameter in their product that doesn't support those characters, so it will fail to run.
They will be working on a fix for this issue and the ETA is at the end of March. The current solution is to use only ASCII characters in the notebook name, in which case everything works fine!
Super UserHi ZJoe,
Most probably it might have to do with environment rather than amount of data.
In your workspace settings, have you set the default environment to be the one that you use to execute interactively?
Thanks for your reply govindarajan_d .
In my workspace, the Set default environment toggle is set to Off. because not all notebooks in my workspace require the same libraries that are included in the environment I use for interactive execution, and it takes longer to start a new session for notebooks that don't need any library if set to On. so I just attach the environment to those specific notebooks.
In your opinion, the environment attached to notebook will be ineffective when running in a scheduled manner?
- govindarajan_d
Hi ZJoe,
The environment selected should have been effective when running in a scheduled manner. That setting is not required, but I thought if the environment is used across all notebooks, we can try to enforce the environment being used just to see if that works.
Two things:
1. Can you select the default environment once and then change it back to test-environment and then see if that works? (Kind of like a reset)
2. In view all runs, you would be able to see the previous runs right. Can you go to the error runs
And see the log information here to see if you could find more information about the error?
Hello govindarajan_d
I make a copy of the notebook for test.
For first thing, I open the notebook, select the default environment, it shows a notification of "Update environment to Workspace Setting", then I change it back to my custom environment, also it shows a notification of "Update environment to py_env_1.3". after that, I set a schedule for it, and when reaching the specified time, the run is failed as before.
For second thing, it seems even stranger and I notice I missed it in the problem description.
All the error runs by schedule only show in the Monitor panel, none of them shows in the View all runs, as if it's not running at all, see below. so again, except the information of "InternalError" shows in the Monitor panel, I can't find more helpful information..
- govindarajan_d
Hi ZJoe,
Ok. We can definitely say that the spark session itself did not start and the problem is in the environment.
I would suggest creating a new environment with the same setting and see if that works ( this is all trial and error method at this point, since the obvious process doesn't seem to happen 😅 ).
Also, I would recommend creating a MS Ticket since this seems to be a bug.