Hello,

I get the following error while trying to stop a spark session in a notebook using: mssparkutils.session.stop()

--------------------------------------------------------------------------- NotebookExit Traceback (most recent call last) File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/session.py:10, in stop() 9 try: ---> 10 nb.exit(constants.STOP_SESSION_REQUEST_EXIT_VAL) 11 finally: File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/notebookHandler.py:66, in SynapseNotebookHandler.exit(self, value) 65 self.exitVal = str(value) ---> 66 raise NotebookExit(value) NotebookExit: __$$_STOP_SESSION_REQUEST_$$__ During handling of the above exception, another exception occurred: Py4JJavaError Traceback (most recent call last) Cell In[23], line 1 ----> 1 mssparkutils.session.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/session.py:12, in stop() 10 nb.exit(constants.STOP_SESSION_REQUEST_EXIT_VAL) 11 finally: ---> 12 sessionHandler.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/sessionHandler.py:9, in SessionHandler.stop(self) 8 def stop(self😞 ----> 9 self.sessionImpl.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command) -> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"😞 File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:169, in capture_sql_exception.<locals>.deco(*a, **kw) 167 def deco(*a: Any, **kw: Any) -> Any: 168 try: --> 169 return f(*a, **kw) 170 except Py4JJavaError as e: 171 converted = convert_exception(e.java_exception) File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE: --> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value)) Py4JJavaError: An error occurred while calling z:com.microsoft.spark.notebook.msutils.impl.SessionImpl.stop. : pdi.jwt.exceptions.JwtLengthException: Expected token [[REDACTED]] to be composed of 2 or 3 parts separated by dots. at pdi.jwt.JwtCore.splitToken(Jwt.scala:190) at pdi.jwt.JwtCore.$anonfun$decodeRawAll$1(Jwt.scala:202) at scala.util.Try$.apply(Try.scala:213) at pdi.jwt.JwtCore.decodeRawAll(Jwt.scala:201) at pdi.jwt.JwtCore.decodeRawAll$(Jwt.scala:201) at pdi.jwt.Jwt$.decodeRawAll(JwtPureScala.scala:19) at mssparkutils.env$.extractField(env.scala:22) at mssparkutils.env$.getPoolName(env.scala:82) at com.microsoft.spark.notebook.msutils.impl.SessionImpl$.stop(SessionImpl.scala:35) at com.microsoft.spark.notebook.msutils.impl.SessionImpl.stop(SessionImpl.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:829)

Does anyone know where the issue may come from ?

Thank you,

Amnay

Forum Discussion

mssparkutils credentials library in notebook

mssparkutils inside python package

mssparkutils run notebook across workspaces

mssparkutils catch exception

Your Compute session is disconnected