%run magic works manually but fails in pipeline when using runMultiple -cross kernel execution error

[mhosseini]

Hi everyone,

It was recently announced that %run magic is now supported for Python notebooks in Fabric, so I started using it to keep my common functions in a separate notebook.

My setup is as follows:

• I have a common notebook that contains shared functions.
• I have several processing notebooks that use those functions.
• I have an orchestrator notebook that runs everything using notebookutils.notebook.runMultiple().

In the orchestrator notebook, I first run the common notebook, and then execute the rest of the notebooks.

When I run the notebooks manually, everything works fine.

However, when the orchestrator notebook is executed from a pipeline, the cell that runs runMultiple() fails with the following error:

Notebook execution failed at Notebook service with http status code - '200',
please check the Run logs on Notebook, additional details -

Error name - Exception
Error value - NBS request failed: 400 - Cross kernel type execution is not supported in %run magic.
Child notebook kernel: python3.11, Current notebook kernel: .

It seems the pipeline execution is triggering a cross-kernel execution issue, but all notebooks are Python notebooks.

Has anyone run into this issue when using %run together with notebookutils.notebook.runMultiple() in a pipeline?

Any suggestions on how to resolve this would be greatly appreciated.

Thanks!

[mhosseini · Accepted solution]

I've raised a support ticket with Microsoft, and they have confirmed this as a bug. They are working on a fix for a future release, though no ETA has been provided at this time.

[SteveCarlsberg2]

Same issue here! Running manually is fine, but running the notebook from a pipeline fails. I'm using python 3.11 version in all notebooks.

[v-achippa]

Hi @mhosseini,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @Murtaza_Ghafoor and @deborshi_nag for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @mhosseini,

We wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[mhosseini]

Hi @v-achippa ,
Unfortunately I don't have a feasible solution so far. And personally I think this is a bug that Microsoft should resolve. %run in python notebooks is not production ready.

[v-achippa]

Hi @mhosseini,

Thank you for the response. I request you to please raise a support ticket for further assistance

Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

Thank you for being part of Microsoft Fabric Community.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @mhosseini,

As we haven’t heard back from you, we wanted to kindly follow up to check if you have raised the support ticket?

Thank you for being part of Microsoft Fabric Community.

Thanks and regards,

Anjan Kumar Chippa

[mhosseini]

Hi @v-achippa , I just created a ticket.

[deborshi_nag]

Hello @mhosseini 

I have run an orchestrator notebook that runs other notebooks using notebookutils.notebook.runMultiple() where the first cell in each notebook is the %run command. I have run the orchestrator notebook in a pipeline, and it works fine. 

The error message suggests a cross-kernel issue. Mixing Spark + Python notebooks in the same orchestration graph is not supported and commonly results in error. 

In my workspace I have used Pyspark notebooks for both the orchestrator as well as the child notebooks - the environments used in each is the workspace default. 

Here's the code I used for my test. 

Orchestrator notebook:

from notebookutils import notebook

# Define the DAG
DAG = {
    "activities": [
        {
            "name": "extract_customers",
            "path": "nb_extract_customers",           # notebook item path/name
            "timeoutPerCellInSeconds": 120,
            "args": {"rows": 1000}
        },
        {
            "name": "extract_products",
            "path": "nb_extract_products",
            "timeoutPerCellInSeconds": 120,
            "args": {"rows": 5000},
            "dependencies": ["extract_customers"]
        },
        {
            "name": "products_table",
            "path": "nb_build_product_table",
            "timeoutPerCellInSeconds": 300,
            "retry": 1,
            "retryIntervalInSeconds": 10,
            "dependencies": ["extract_products"]      # wait for products first
        }
    ],
    "timeoutInSeconds": 1800,
    "concurrency": 3
}

# Kick off the DAG; options can enable a DAG visualization and live progress
result = notebook.runMultiple(
    DAG,
    {"displayDAGViaGraphviz": True}  # optional visualization
)

# 'result' is typically a structured object; you can inspect/log it
print(result)

Child notebook:

%run nb_common_functions

# Parameters cell (mark as "Toggle parameter cell" in UI)
rows = 1000  # default value, can be overridden by args

# ... your logic here ...
result = f"customers_ok_{rows}"

hello("customers")

from notebookutils import notebook
notebook.exit(result)

[mhosseini]

Hi @deborshi_nag ,

Thank you for your response. I'm using python notebook. I already have spark notebooks and using the same method works. But Python notebooks have this issue.

[deborshi_nag]

Thanks @mhosseini, confirms my point - Mixing Spark + Python notebooks in the same orchestration graph is not supported, which is also what the error message indicates. 

In practice:

• Controller / orchestrator - must be Spark
• Child notebooks - should also be Spark

[mhosseini]

I'm not mixing those runtimes, all my notebooks in this setup are using python runtime.

I just maant that I have a similar orchestration that is using spark from end to end, it has %run and runMultiple the same way, and it is working as expected.

[deborshi_nag]

Hey @mhosseini 

Microsoft refers to NotebookUtils (formerly MSSparkUtils) as the standard package for chaining notebooks, managing files, secrets, and more. It is available for PySpark, Scala, and SparkR notebooks, and can also be invoked from pipelines. The latest enhancements target Spark 3.4+ (Fabric Runtime v1.2+), operating within the notebookutils namespace, where run() and runMultiple() are designed to function. Hence you need the orchestrator to be a Spark notebook. 

NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

The reason manual runs work is that Fabric reuses your current Spark session, whereas pipelines always initiate a new session that is strictly validated.

When you select Run in the notebook interface, the editor connects to an active Spark session with the kernel and libraries already loaded. runMultiple() inherits this session, allowing %run to execute within the same process.

For runs triggered by pipelines, there is no pre-existing session. Fabric creates a new execution environment and attaches a kernel based on the notebook’s metadata.

[Murtaza_Ghafoor]

@mhosseini 
Why this happening:

The error happens because pipelines run notebooks in separate kernels, and %run does not support cross-kernel execution. 

Temporary Solution:

Make sure every notebook that uses shared functions runs %run itself, instead of running it only in the orchestrator.

Example inside each notebook:

%run "./CommonFunctions"

But this still may not be ideal for pipelines.

Best Possible Approach:

Convert the Common Notebook to a Library

Fabric supports creating workspace libraries.

You can:

1. Package your shared functions as a Python module
2. Upload it as a workspace library
3. Import it in notebooks

This is the most scalable enterprise approach.

Best Architecture for Fabric Notebooks

Instead of:

Common Notebook
      ↑
   %run
      ↑
Processing Notebooks
      ↑
Orchestrator Notebook

Use

Python Module / Library
        ↑
Processing Notebooks
        ↑
Orchestrator Notebook
        ↑
Pipeline

This will remove the dependency on kernel. 

If this helps, ✓ Mark as Kudos | Mark as Solution| Help Others

[mhosseini]

Hi @Murtaza_Ghafoor 

Thank you for your response. I should remphesis on the fact that I am using python notebooks, unlike spark environemtns which gives us a place to upload libraries, python does not provide such feature, would you let me know if there's a way to upload a custom library in python notebooks?

Also I tried several combinations:

1. Orchestrator runs common notebook & Processing runs common notebook > Cross runtime failure

2. Orchestrator does not run common notebook & Processing runs common notebook > Cross runtime failure

3. Orchestrator runs common notebook & Processingdoes not run common notebook > common functions are not available in processing notebook failure

[Murtaza_Ghafoor]

@mhosseini 
Since you are using python notebooks so this option is not available but there is one workaround you may try this is to store your shared functions in a .py file inside Lakehouse Files or Workspace Files, and then import that file in your notebooks using sys.path.

For example:

import sys

sys.path.append('/lakehouse/default/Files/utils')

import common_functions

With this approach, the notebooks load the shared functions directly from the file system rather than relying on the same kernel session. Because of that, it works consistently for both manual notebook runs and pipeline executions.

If this helps, ✓ Mark as Kudos | Mark as Solution| Help Others

[mhosseini]

Thank you @Murtaza_Ghafoor this is the best suggestion so far. It works if I add the lakehouse containing the python file as the default lakehouse. is there a way to read it from a single location whether it is attached or not? I don't want to add a huge piece of code to all the notebooks for this.