Forum Discussion

Dinosauris's avatar
Dinosauris
Helper I
2 years ago
Solved

Running a Fabric Notebook as Spark Job.

Hello,   I am interested in converting a Fabric notebook that I’ve created, containing multiple transformation steps, into a Spark job. However, I’m struggling to find comprehensive documentation o...
data engineering
  • Dinosauris's avatar
    Dinosauris
    2 years ago

    Hello!

     

    In the meantime we decided to run the Fabric notebooks via pipelines for the use case.

     

    Thank you anyways for the help! 🙂

Dinosauris's avatar
Dinosauris
Helper I
2 years ago

Thanks for the quick reply! 

 

Finally, I did two things, I added the following code at the beginning of my notebook to start the spark session:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
.appName("MySparkJob") \
.getOrCreate()

 

And the list code line is stopping the spark session:

spark.stop()

 

However, I get this error message when trying to run the spark job: Execution is not supported for Spark Job Definitions that do not have content.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Dinosauris 
Apologies for the issue you have been facing.
You can also run your notebooks using Notebook activity in pipelines. Pipeline has more powerful features. And some mssparkutils APIs are not supported in Spark Job Definition.
For more information please refer to this link:
Notebook activity - Microsoft Fabric | Microsoft Learn
Scheduling Notebooks in Microsoft Fabric + Reading JSON from Dynamic File Paths (youtube.com)

Hope this helps. Please let me know if you have any further questions.


  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Dinosauris 
    Apologies for the delay in response. 

    Please go ahead and raise a support ticket to reach our support team: Link 

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

    Thanks.