spark NameError when using an imported function needing SparkContext

When you develop locally, your spark.py file explicitly creates a SparkSession using SparkSession.builder. This works because you control the full Python environment and are expected to instantiate Spark manually.

However, in Microsoft Fabric Notebooks, a SparkSession is already created and provided implicitly by the runtime. You can access it simply via the spark object, but you should not re-instantiate or create a new SparkSession.

Don't define your own spark.py module

Rename your module to something else like spark_utils.py or file_io.py. This avoids shadowing the built-in spark object.

Use the implicit spark from Fabric

Remove the following from your code entirely:

CopyEdit
from pyspark.sql import SparkSession

spark = SparkSession.builder \
        .appName("GlobalSpark") \
        .master("local[*]") \
        .getOrCreate()

Instead, in your utility function, rely on the pre-provided spark:

CopyEdit
def get_file(outputFilePath: str):
    df = spark.read.option("multiline", "true").json(outputFilePath)
    return df

Avoid import spark altogether

If you must package utilities, structure it like this:

CopyEdit
# file_io.py
def get_file(outputFilePath: str):
    from pyspark.sql import SparkSession
    spark = SparkSession.getActiveSession()
    if spark is None:
        raise RuntimeError("No active SparkSession found. This function must be run within a Spark environment.")
    df = spark.read.option("multiline", "true").json(outputFilePath)
    return df

But in Fabric, SparkSession.getActiveSession() should return the running session just fine.

Please mark this post as solution if it helps you. Appreciate Kudos.