mssparkutils run notebook across workspaces

[Anonymous]

Greetings!
I am trying to use the mssparkutils.notebook.run() function to run a notebook in a different workspace. I get the same error for both these variants:

mssparkutils.notebook.run(path="notebook_name", timeout_seconds= 90, arguments={}, workspace='workspace-ID-4444-4444-what4ever')

mssparkutils.notebook.run(path="notebook_name", timeout_seconds= 90, arguments={}, workspace='workspace name')

The error says, among a lot of noise:

Py4JJavaError: An error occurred while calling z:mssparkutils.notebook.run. : com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for gold_dimentity failed with exception: Not able to get workspace id by name 33333333-7777-4444-aaaa-iiiiiiiiiii with exception - Expected collection but got JObject[...]

I have no idea what to do next. I kindly appreciate any help.
Thanks!
(@v-nikhilan-msft)

[Anonymous]

Hi @Anonymous 
Thanks for using Fabric Community.
I tried to repro the scenario and was successful in running the notebook.

You can give the name of the workspace or the id of the workspace as the input for workspace parameter. Try using any one. 

Please give the correct id of the workspace . You can fetch it from the URL. 
https://msit.powerbi.com/groups/9aa56487-7312-4340-ab81-690afd6b3768/
The string in bold letters would be the workspace id .

Hope this helps. Try the above steps and let me know if the issue still persists. Please paste the entire error output log here if you still get the same error.

Thanks

[Anonymous]

Hm, it works in my personal account's trial capacity. The workspaces that don't work are in an organizational F2/F4 capacity, connected to git and part of a development pipeline. I'm also not a capacity admin there, but for the workspaces.

Anyways, here's the full error message:

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
Cell In[11], line 1
----> 1 mssparkutils.notebook.run(path="00_Broken_Notebook", timeout_seconds= 90, arguments={}, workspace='Fabric 1_Development')

File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/notebook.py:13, in run(path, timeout_seconds, arguments, workspace)
     12 def run(path, timeout_seconds=90, arguments={}, workspace=""):
---> 13     exit_val = nb.run(path, timeout_seconds, arguments, workspace)
     15     if exit_val == constants.STOP_SESSION_REQUEST_EXIT_VAL:
     16         nb.exit(exit_val)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/notebookHandler.py:56, in SynapseNotebookHandler.run(self, path, timeout_seconds, arguments, workspace)
     53 def run(self, path, timeout_seconds=90, arguments={}, workspace=""):
     54     self.check_types([(path, string_types), (timeout_seconds, int), (arguments, dict)])
---> 56     j_notebook_run_result = self.notebookutils.run(
     57         path, timeout_seconds, arguments, workspace, self.runtimeUtils.getContextAsJMap())
     58     exit_val = j_notebook_run_result.exitVal()
     59     self._display_snapshot(j_notebook_run_result)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
   1315 command = proto.CALL_COMMAND_NAME +\
   1316     self.command_header +\
   1317     args_command +\
   1318     proto.END_COMMAND_PART
   1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
   1322     answer, self.gateway_client, self.target_id, self.name)
   1324 for temp_arg in temp_args:
   1325     temp_arg._detach()

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py:190, in capture_sql_exception.<locals>.deco(*a, **kw)
    188 def deco(*a: Any, **kw: Any) -> Any:
    189     try:
--> 190         return f(*a, **kw)
    191     except Py4JJavaError as e:
    192         converted = convert_exception(e.java_exception)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling z:mssparkutils.notebook.run.
: com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for 00_Broken_Notebook failed with exception: Not able to get workspace id by name Fabric 1_Development with exception - Expected collection but got JObject(List((value,JArray(List(JObject(List((id,JString(XXXX)), (displayName,JString(XXXX)), [List of capacity/workspace names and IDs]))))))))) for root JObject(List((value,JArray(List(JObject(List((id,JString(XXXX)), (displayName,JString(XXXX)), (description,JString()), (type,JString(Workspace)))), JObject(List((id,JString([List of capacity/workspace names and IDs]))))))))) and mapping Workspace[][Workspace, Workspace].
	at com.microsoft.spark.notebook.msutils.impl.notebook.IMSNotebookProvider.get(IMSNotebookProvider.scala:47)
	at com.microsoft.spark.notebook.msutils.impl.notebook.IMSNotebookProvider.get$(IMSNotebookProvider.scala:21)
	at com.microsoft.spark.notebook.msutils.impl.notebook.TridentNotebookProvider.get(TridentNotebookProvider.scala:14)
	at com.microsoft.spark.notebook.msutils.impl.MSNotebookUtilsImpl._run(MSNotebookUtilsImpl.scala:96)
	at com.microsoft.spark.notebook.msutils.impl.MSNotebookUtilsImpl.run(MSNotebookUtilsImpl.scala:165)
	at mssparkutils.notebook$.run(notebook.scala:53)
	at mssparkutils.notebook.run(notebook.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.Exception: Not able to get workspace id by name Fabric 1_Development with exception - Expected collection but got JObject([same as above]))))))))) for root JObject(List((value,JArray(List(JObject(List((id,JString([same as above]))))))))) and mapping Workspace[][Workspace, Workspace]
	at com.microsoft.spark.notebook.msutils.impl.artifact.MSArtifactUtilsImpl.getWorkspaceId(MSArtifactUtilsImpl.scala:213)
	at com.microsoft.spark.notebook.msutils.impl.notebook.TridentNotebookProvider.getWorkspaceId(TridentNotebookProvider.scala:43)
	at com.microsoft.spark.notebook.msutils.impl.notebook.IMSNotebookProvider.get(IMSNotebookProvider.scala:35)
	... 17 more

[Anonymous]

Hi @Anonymous 
You are getting this error because you are using Spark Runtime1.1 (spark3.3) in the Notebook. Please change to Runtime 1.2 and run the code.

Hope this helps. Please let me know if the issue still persists.

[Anonymous · Accepted solution]

Well, I kept the Spark Version low, because otherwise I had minReaderVersion compatibility issues. I will try it, but this minReaderVersion stuff broke pretty much everything.

Edit: Nope, still doesn't work, unfortunately, will look for a fix, but the issue in this thread is the minor one compared to the minReaderVersion.

Py4JJavaError: An error occurred while calling o4423.throwExceptionIfHave.
: com.microsoft.spark.notebook.msutils.NotebookExecutionException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures.