%run magic works manually but fails in pipeline when using runMultiple -cross kernel execution error

Same issue here! Running manually is fine, but running the notebook from a pipeline fails. I'm using python 3.11 version in all notebooks.

mhosseini 
Why this happening:

The error happens because pipelines run notebooks in separate kernels, and %run does not support cross-kernel execution. 

Temporary Solution:

Make sure every notebook that uses shared functions runs %run itself, instead of running it only in the orchestrator.

Example inside each notebook:

%run "./CommonFunctions"

But this still may not be ideal for pipelines.

Best Possible Approach:

Convert the Common Notebook to a Library

Fabric supports creating workspace libraries.

You can:

1. Package your shared functions as a Python module
2. Upload it as a workspace library
3. Import it in notebooks

This is the most scalable enterprise approach.

Best Architecture for Fabric Notebooks

Instead of:

Common Notebook
      ↑
   %run
      ↑
Processing Notebooks
      ↑
Orchestrator Notebook

Use

Python Module / Library
        ↑
Processing Notebooks
        ↑
Orchestrator Notebook
        ↑
Pipeline

This will remove the dependency on kernel. 

If this helps, ✓ Mark as Kudos | Mark as Solution| Help Others

Hello mhosseini 

I have run an orchestrator notebook that runs other notebooks using notebookutils.notebook.runMultiple() where the first cell in each notebook is the %run command. I have run the orchestrator notebook in a pipeline, and it works fine. 

The error message suggests a cross-kernel issue. Mixing Spark + Python notebooks in the same orchestration graph is not supported and commonly results in error. 

In my workspace I have used Pyspark notebooks for both the orchestrator as well as the child notebooks - the environments used in each is the workspace default. 

Here's the code I used for my test. 

Orchestrator notebook:

from notebookutils import notebook

# Define the DAG
DAG = {
    "activities": [
        {
            "name": "extract_customers",
            "path": "nb_extract_customers",           # notebook item path/name
            "timeoutPerCellInSeconds": 120,
            "args": {"rows": 1000}
        },
        {
            "name": "extract_products",
            "path": "nb_extract_products",
            "timeoutPerCellInSeconds": 120,
            "args": {"rows": 5000},
            "dependencies": ["extract_customers"]
        },
        {
            "name": "products_table",
            "path": "nb_build_product_table",
            "timeoutPerCellInSeconds": 300,
            "retry": 1,
            "retryIntervalInSeconds": 10,
            "dependencies": ["extract_products"]      # wait for products first
        }
    ],
    "timeoutInSeconds": 1800,
    "concurrency": 3
}

# Kick off the DAG; options can enable a DAG visualization and live progress
result = notebook.runMultiple(
    DAG,
    {"displayDAGViaGraphviz": True}  # optional visualization
)

# 'result' is typically a structured object; you can inspect/log it
print(result)

Child notebook:

%run nb_common_functions

# Parameters cell (mark as "Toggle parameter cell" in UI)
rows = 1000  # default value, can be overridden by args

# ... your logic here ...
result = f"customers_ok_{rows}"

hello("customers")

from notebookutils import notebook
notebook.exit(result)

Hi mhosseini,

Thank you for reaching out to Microsoft Fabric Community.

Thank you Murtaza_Ghafoor and deborshi_nag for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa