topic Re: Running a Fabric Notebook as Spark Job. in Data Engineering

Running a Fabric Notebook as Spark Job.

Dinosauris — Wed, 21 Feb 2024 08:46:19 GMT

Hello,

I am interested in converting a Fabric notebook that I’ve created, containing multiple transformation steps, into a Spark job. However, I’m struggling to find comprehensive documentation on the additional code I need to include (such as building a Spark session) and how to create a reference file if necessary (along with the required code). Additionally, I’m unsure about the modifications needed in my notebook to enable downloading it as a .py file and running it as a Spark job.

The motivation behind this transition is that I want the transformation code to execute every 2 hours, and from what I’ve read, using Spark jobs may offer better performance for this use case.

I would greatly appreciate it if someone could provide an example of how to achieve this.

Thank you in advance! 😊

Re: Running a Fabric Notebook as Spark Job.

Anonymous — Wed, 21 Feb 2024 08:51:49 GMT

Hi @Dinosauris
Thanks for using Fabric Community.
Please refer to these documents:
https://learn.microsoft.com/en-us/fabric/data-engineering/create-spark-job-definition
https://www.red-gate.com/simple-talk/databases/sql-server/bi-sql-server/using-spark-jobs-for-multiple-lakehouse-maintenance-in-microsoft-fabric/

Hope this helps. Please let me know if you have any further questions.

Re: Running a Fabric Notebook as Spark Job.

Dinosauris — Wed, 21 Feb 2024 09:11:54 GMT

Thanks for the quick reply!

Finally, I did two things, I added the following code at the beginning of my notebook to start the spark session:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
.appName("MySparkJob") \
.getOrCreate()

And the list code line is stopping the spark session:

spark.stop()

However, I get this error message when trying to run the spark job: Execution is not supported for Spark Job Definitions that do not have content.

Re: Running a Fabric Notebook as Spark Job.

Anonymous — Wed, 21 Feb 2024 09:23:23 GMT

Hi @Dinosauris
Apologies for the issue you have been facing.
You can also run your notebooks using Notebook activity in pipelines. Pipeline has more powerful features. And some mssparkutils APIs are not supported in Spark Job Definition.
For more information please refer to this link:
Notebook activity - Microsoft Fabric | Microsoft Learn
Scheduling Notebooks in Microsoft Fabric + Reading JSON from Dynamic File Paths (youtube.com)

Hope this helps. Please let me know if you have any further questions.

Re: Running a Fabric Notebook as Spark Job.

HimanshuS-msft — Wed, 21 Feb 2024 18:14:03 GMT

Hello @Dinosauris ,

Just to provide a bit more clarity here . Since the transformation is already there in the notebbok , you can add that to a pipeline ( as @Anonymous called out above ) or you can scheduled from the notebook pane itself . The below snapshot should help .

Thanks
Himanshu

Re: Running a Fabric Notebook as Spark Job.

Dinosauris — Thu, 22 Feb 2024 08:20:08 GMT

Hello Himanshu,

thanks for the info. I am aware of this and this is my backup option (and will probably be the solution of the problem). However, I still wanted to understand why I cannot make the Notebook work using Spark Job. I have a lot of display() functions in there and don't know whether this could be an issue. No mssparkutils APIs are called.

Thanks & BR 🙂

Re: Running a Fabric Notebook as Spark Job.

Anonymous — Tue, 27 Feb 2024 15:56:55 GMT

Hi @Dinosauris
Apologies for the delay in response.

Please go ahead and raise a support ticket to reach our support team: Link

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks.

Re: Running a Fabric Notebook as Spark Job.

Dinosauris — Fri, 08 Mar 2024 06:28:35 GMT

Hello!

In the meantime we decided to run the Fabric notebooks via pipelines for the use case.

Thank you anyways for the help! 🙂