Hi Community!
Disclaimer: No idea if this is the correct place for such a topic but I didn't know where to put it exactly.
I think the following situation arised with the May 2024 Microsoft Fabric Update - can't be sure though because the processes didn't run continuously but the following problem definitely didn't arise in April.
We have a Microsoft Fabric workspace with 4 notebooks that we orchestrate with a seperate orchestration-notebook via mssparkutils to run the notebooks in parallel. The orchestration-notebook is then called in a pipeline and is bound to an environment that runs on Fabric Runtime 1.1. This is the case because the 4 notebooks make use of the cdm connector for spark, that is only available in runtime 1.1.
The orchestration-notebook looks like that:
from notebookutils import mssparkutils
spark.conf.set('spark.sql.parquet.vorder.enabled', 'true')
DAG = {
"activities": [
{
"name": "Notebook1",
"path": "Notebook1",
"timeoutPerCellInSeconds": 300,
"args": {"useRootDefaultLakehouse": True}
},
{
"name": "Notebook2",
"path": "Notebook2",
"timeoutPerCellInSeconds": 300,
"args": {"useRootDefaultLakehouse": True}
},
{
"name": "Notebook3",
"path": "Notebook3",
"timeoutPerCellInSeconds": 300,
"args": {"useRootDefaultLakehouse": True}
},
{
"name": "Notebook4",
"path": "Notebook4",
"timeoutPerCellInSeconds": 300,
"args": {"useRootDefaultLakehouse": True}
}
]
}
mssparkutils.notebook.runMultiple(DAG)
The above setting worked just fine but as described approximately since the May 2024 update we get the following error message when executing the orchestration-notebook via a pipeline-call or as a sheduled notebook:
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook1 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook2 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook3 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for Notebook4 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...
{'Notebook1': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook1 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')},
'Notebook2': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook2 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')},
'Notebook3': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook3 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')},
'Notebook4': {'exitVal': '',
'exception': Exception('Fetch notebook content for Notebook4 failed with exception: Request to fetchNotebookContent failed with status code 400 error No kernel id and run id...')}}
Note: Executing the notebook directly via run-button in the notebook workes fine!
