Possible Identity Problem with mssparkutils.notebook.runMultiple in Fabric Runtime 1.1

[Hofpower]

Hi Community!

Disclaimer: No idea if this is the correct place for such a topic but I didn't know where to put it exactly.

I think the following situation arised with the May 2024 Microsoft Fabric Update - can't be sure though because the processes didn't run continuously but the following problem definitely didn't arise in April.

We have a Microsoft Fabric workspace with 4 notebooks that we orchestrate with a seperate orchestration-notebook via mssparkutils to run the notebooks in parallel. The orchestration-notebook is then called in a pipeline and is bound to an environment that runs on Fabric Runtime 1.1. This is the case because the 4 notebooks make use of the cdm connector for spark, that is only available in runtime 1.1. 

The orchestration-notebook looks like that: 

from notebookutils import mssparkutils
spark.conf.set('spark.sql.parquet.vorder.enabled', 'true')

DAG = {
  "activities": [
        {
            "name": "Notebook1", 
            "path": "Notebook1", 
            "timeoutPerCellInSeconds": 300, 
            "args": {"useRootDefaultLakehouse": True}
        },
        {
            "name": "Notebook2",
            "path": "Notebook2", 
            "timeoutPerCellInSeconds": 300, 
            "args": {"useRootDefaultLakehouse": True} 
        },
        {
            "name": "Notebook3", 
            "path": "Notebook3", 
            "timeoutPerCellInSeconds": 300,
            "args": {"useRootDefaultLakehouse": True}
        },
        {
            "name": "Notebook4", 
            "path": "Notebook4", 
            "timeoutPerCellInSeconds": 300,
            "args": {"useRootDefaultLakehouse": True} 
        }
    ]
}
mssparkutils.notebook.runMultiple(DAG)

The above setting worked just fine but as described approximately since the May 2024 update we get the following error message when executing the orchestration-notebook via a pipeline-call or as a sheduled notebook:

com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook1 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook2 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook3 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook4 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
{'Notebook1': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook1 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')},
'Notebook2': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook2 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')},
'Notebook3': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook3 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')},
'Notebook4': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook4 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')}}

Note: Executing the notebook directly via run-button in the notebook workes fine! 

[v-cboorla-msft]

Hi @Hofpower 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[Hofpower]

Hi @v-cboorla-msft 

Thanks for the quick response. We raised a support ticket as suggested, the id is 2406060050004986.

BR