topic Can't stop spark session using mssparkutils in Data Engineering

Can't stop spark session using mssparkutils

akanane — Mon, 20 Nov 2023 10:46:52 GMT

Hello,

I get the following error while trying to stop a spark session in a notebook using: mssparkutils.session.stop()

--------------------------------------------------------------------------- NotebookExit Traceback (most recent call last) File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/session.py:10, in stop() 9 try: ---> 10 nb.exit(constants.STOP_SESSION_REQUEST_EXIT_VAL) 11 finally: File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/notebookHandler.py:66, in SynapseNotebookHandler.exit(self, value) 65 self.exitVal = str(value) ---> 66 raise NotebookExit(value) NotebookExit: __$$_STOP_SESSION_REQUEST_$$__ During handling of the above exception, another exception occurred: Py4JJavaError Traceback (most recent call last) Cell In[23], line 1 ----> 1 mssparkutils.session.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/session.py:12, in stop() 10 nb.exit(constants.STOP_SESSION_REQUEST_EXIT_VAL) 11 finally: ---> 12 sessionHandler.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/sessionHandler.py:9, in SessionHandler.stop(self) 8 def stop(self😞 ----> 9 self.sessionImpl.stop() File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command) -> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"😞 File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:169, in capture_sql_exception.<locals>.deco(*a, **kw) 167 def deco(*a: Any, **kw: Any) -> Any: 168 try: --> 169 return f(*a, **kw) 170 except Py4JJavaError as e: 171 converted = convert_exception(e.java_exception) File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE: --> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value)) Py4JJavaError: An error occurred while calling z:com.microsoft.spark.notebook.msutils.impl.SessionImpl.stop. : pdi.jwt.exceptions.JwtLengthException: Expected token [[REDACTED]] to be composed of 2 or 3 parts separated by dots. at pdi.jwt.JwtCore.splitToken(Jwt.scala:190) at pdi.jwt.JwtCore.$anonfun$decodeRawAll$1(Jwt.scala:202) at scala.util.Try$.apply(Try.scala:213) at pdi.jwt.JwtCore.decodeRawAll(Jwt.scala:201) at pdi.jwt.JwtCore.decodeRawAll$(Jwt.scala:201) at pdi.jwt.Jwt$.decodeRawAll(JwtPureScala.scala:19) at mssparkutils.env$.extractField(env.scala:22) at mssparkutils.env$.getPoolName(env.scala:82) at com.microsoft.spark.notebook.msutils.impl.SessionImpl$.stop(SessionImpl.scala:35) at com.microsoft.spark.notebook.msutils.impl.SessionImpl.stop(SessionImpl.scala) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:829)

Does anyone know where the issue may come from ?

Thank you,

Amnay

Re: Can't stop spark session using mssparkutils

v-cboorla-msft — Mon, 20 Nov 2023 12:15:02 GMT

Hi @akanane

Thanks for using Microsoft Fabric Community.

Apologies for the issue that you are facing.

As I understand that you are getting an error while trying to stop a spark session in a notebook using: mssparkutils.session.stop().

To resolve this issue, you need to ensure that the JWT (JSON Web Token) is correctly formatted and contains the necessary information for authentication.

Here are a few steps you can take to troubleshoot and fix the issue:

Check the JWT token: Verify that the JWT token you are using is valid and properly formatted. Make sure it contains all the required parts (header, payload, signature) separated by dots.
Regenerate the JWT token: If you suspect that the current JWT token is invalid or expired, try regenerating a new token from your authentication provider.
Verify authentication configuration: Double-check your authentication configuration settings to ensure they are correctly set up. Make sure you have provided all the necessary information for authentication, such as client ID, client secret, audience, etc.
Check dependencies: Ensure that all required dependencies and libraries for handling JWT tokens are properly installed and up to date.

I hope this helps. Please do let us know if you have any further questions.

Re: Can't stop spark session using mssparkutils

akanane — Mon, 20 Nov 2023 12:44:26 GMT

Is there any documentation regarding the fix that you provided?

I was following the section: Introduction to Microsoft Spark utilities - Azure Synapse Analytics | Microsoft Learn. There is no mention of JWT tokens..

Re: Can't stop spark session using mssparkutils

v-cboorla-msft — Mon, 27 Nov 2023 11:24:51 GMT

Hi @akanane

Apologies for the delay in response.

Currently mssparkutils supports only the below utilities.

fs:Utility for filesystem operations in Fabric
notebook:Utility for notebook operations (e.g, chaining Fabric notebooks together)
credentials:Utility for obtaining credentials (tokens and keys) for Fabric resources
lakehouse:Utility for lakehouse operations (e.g, create, delete, update, list lakehouse)

It doesn't support the session utility, this is the reason why you are facing an issue while running the code "mssparkutils.session.stop()".

You can use spark.stop() :

If you have spark as a SparkSession object then call spark.stop() to stop the session. This ensures that resources are properly released and the Spark application terminates gracefully.

I hope this helps. Please do let us know if you have any further questions.

Re: Can't stop spark session using mssparkutils

v-cboorla-msft — Wed, 29 Nov 2023 16:44:23 GMT

Hi @akanane

Apologies for the issue you have been facing. Currently mssparkutils.session.stop() does not support in Fabric for now. The feature is releasing. We will have to wait until mssparkutils environment utils is ready.

Appreciate your patience.

Hope this helps. Please let us know if you have any further questions. Glad to help.

Re: Can't stop spark session using mssparkutils

v-cboorla-msft — Fri, 01 Dec 2023 11:48:17 GMT

Hi @akanane

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.

In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Can't stop spark session using mssparkutils

akanane — Fri, 01 Dec 2023 12:03:11 GMT

Hello,

Thank you for your answer, spark.stop() works 🙂 (but we don't see the session stopped in the UI)

Re: Can't stop spark session using mssparkutils

v-cboorla-msft — Fri, 01 Dec 2023 12:09:17 GMT

Hi @akanane

Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

Re: Can't stop spark session using mssparkutils

Anonymous — Sun, 07 Apr 2024 11:31:58 GMT

With this command, session does not stop.

Re: Can't stop spark session using mssparkutils

Anonymous — Sun, 07 Apr 2024 11:34:39 GMT

I need to complete the run of pipeline within 15 mins but unfortunetly, most of the time is taken to start notebook sessions. To save time, exploring options to stop session. Please help with fabric option supported as of today.