Forum Discussion

akanane's avatar
akanane
Frequent Visitor
2 years ago
Solved

Can't stop spark session using mssparkutils

 
Hello,

 

I get the following error while trying to stop a spark session in a notebook using: mssparkutils.session.stop()

 

--------------------------------------------------------------------------- NotebookExit Traceback (most recent call last) File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/session.py:10, in stop() 9 try: ---> 10 nb.exit(constants.STOP_SESSION_REQUEST_EXIT_VAL) 11 finally: File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/notebookHandler.py:66, in SynapseNotebookHandler.exit(self, value) 65 self.exitVal = str(value) ---> 66 raise NotebookExit(value) NotebookExit: __$$_STOP_SESSION_REQUEST_$$__ During handling of the above exception, another exception occurred: Py4JJavaError Traceback (most recent call last) Cell In[23], line 1 ----> 1 mssparkutils.session.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/session.py:12, in stop() 10 nb.exit(constants.STOP_SESSION_REQUEST_EXIT_VAL) 11 finally: ---> 12 sessionHandler.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/sessionHandler.py:9, in SessionHandler.stop(self) 8 def stop(self😞 ----> 9 self.sessionImpl.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command) -> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"😞 File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:169, in capture_sql_exception.<locals>.deco(*a, **kw) 167 def deco(*a: Any, **kw: Any) -> Any: 168 try: --> 169 return f(*a, **kw) 170 except Py4JJavaError as e: 171 converted = convert_exception(e.java_exception) File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE: --> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value)) Py4JJavaError: An error occurred while calling z:com.microsoft.spark.notebook.msutils.impl.SessionImpl.stop. : pdi.jwt.exceptions.JwtLengthException: Expected token [[REDACTED]] to be composed of 2 or 3 parts separated by dots. at pdi.jwt.JwtCore.splitToken(Jwt.scala:190) at pdi.jwt.JwtCore.$anonfun$decodeRawAll$1(Jwt.scala:202) at scala.util.Try$.apply(Try.scala:213) at pdi.jwt.JwtCore.decodeRawAll(Jwt.scala:201) at pdi.jwt.JwtCore.decodeRawAll$(Jwt.scala:201) at pdi.jwt.Jwt$.decodeRawAll(JwtPureScala.scala:19) at mssparkutils.env$.extractField(env.scala:22) at mssparkutils.env$.getPoolName(env.scala:82) at com.microsoft.spark.notebook.msutils.impl.SessionImpl$.stop(SessionImpl.scala:35) at com.microsoft.spark.notebook.msutils.impl.SessionImpl.stop(SessionImpl.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:829)
 

Does anyone know where the issue may come from ?

Thank you,

 

Amnay

data engineering
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi akanane 

     

    Apologies for the issue you have been facing. Currently mssparkutils.session.stop() does not support in Fabric for now. The feature is releasing. We will have to wait until mssparkutils environment utils is ready.

     

    Appreciate your patience.

     

    Hope this helps. Please let us know if you have any further questions. Glad to help.

9 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi akanane 

     

    Thanks for using Microsoft Fabric Community.

    Apologies for the issue that you are facing.

    As I understand that you are getting an error while trying to stop a spark session in a notebook using: mssparkutils.session.stop().

     

    To resolve this issue, you need to ensure that the JWT (JSON Web Token) is correctly formatted and contains the necessary information for authentication.

    Here are a few steps you can take to troubleshoot and fix the issue:

    • Check the JWT token: Verify that the JWT token you are using is valid and properly formatted. Make sure it contains all the required parts (header, payload, signature) separated by dots.
    • Regenerate the JWT token: If you suspect that the current JWT token is invalid or expired, try regenerating a new token from your authentication provider.
    • Verify authentication configuration: Double-check your authentication configuration settings to ensure they are correctly set up. Make sure you have provided all the necessary information for authentication, such as client ID, client secret, audience, etc.
    • Check dependencies: Ensure that all required dependencies and libraries for handling JWT tokens are properly installed and up to date.

    I hope this helps. Please do let us know if you have any further questions.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi akanane 

         

        Apologies for the delay in response.

        Currently mssparkutils supports only the below utilities.

        fs:Utility for filesystem operations in Fabric
        notebook:Utility for notebook operations (e.g, chaining Fabric notebooks together)
        credentials:Utility for obtaining credentials (tokens and keys) for Fabric resources
        lakehouse:Utility for lakehouse operations (e.g, create, delete, update, list lakehouse)

        It doesn't support the session utility, this is the reason why you are facing an issue while running the code "mssparkutils.session.stop()".

        You can use spark.stop() :

        If you have spark as a SparkSession object then call spark.stop() to stop the session. This ensures that resources are properly released and the Spark application terminates gracefully.

         

        I hope this helps. Please do let us know if you have any further questions.