topic F2 Capacity. spark.stop() at the end of the notebook but still running after this command in Data Engineering

F2 Capacity. spark.stop() at the end of the notebook but still running after this command

DebbieE — Fri, 09 Aug 2024 08:18:40 GMT

I have this at the end of a notebook (And I am in F2 Capacity

# Stop the Spark session

spark.stop()

But I can;t run another notebook afterwards because it thinks it is still running. If I go into Monotor it still says in progress. i have to cancel it manually in Monitor.

I really need the stop command to work. Is there something I am doing wrong?

Re: F2 Capacity. spark.stop() at the end of the notebook but still running after this command

DebbieE — Fri, 09 Aug 2024 13:24:09 GMT

I have 4 notebooks which I want to run in a sequence.

So I join them together and run, It just sticks on the first Notebook and doesnt move to the next notebook

I have

spark.stop()

at the end of each notebook. And I know they only take a couple of minutes. Why isn't it moving on?

Re: F2 Capacity. spark.stop() at the end of the notebook but still running after this command

DebbieE — Fri, 09 Aug 2024 14:00:59 GMT

It seemed to run quickly this time. Maybe it was just a blip. And I have changed the connections to success ones

Re: F2 Capacity. spark.stop() at the end of the notebook but still running after this command

frithjof_v — Sat, 10 Aug 2024 07:34:25 GMT

Thanks for sharing!

I'm curious why you are using spark.stop()?

I'm curious if it is necessary to use it or not.

I didn't think it was necessary in Fabric. Did you find any information about it?

Re: F2 Capacity. spark.stop() at the end of the notebook but still running after this command

frithjof_v — Sat, 10 Aug 2024 15:40:22 GMT

Are you running the notebook manually, or inside a data pipeline? If you are running them inside a data pipeline, perhaps you can add a wait activity between them. Perhaps wait 1 or 2 minutes.

Microsoft Fabric – Notebook session usage explained (And how to save CU’s or billed time) - FourMoo | Fabric | Power BI

spark.stop() - is it needed? : r/MicrosoftFabric (reddit.com)

Are you using Notebook A to trigger Notebook B? Then I think they can share the compute resources. See links below:

Microsoft Spark Utilities (MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

Using runMultiple To Orchastrate Notebook Execution in Microsoft Fabric

You could also try out the mssparkutils.session.stop() instead of spark.stop()

I haven't tried it myself.

Re: F2 Capacity. spark.stop() at the end of the notebook but still running after this command

Anonymous — Tue, 13 Aug 2024 01:12:12 GMT

Hi DebbieE,

In fact, you not need to run these processing one by one. You can invoke other notebook codesnip or directly invoke/reference other notebooks by NotebookUtils functions:

NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

If you want stop all the cost of fabric capacity resource usage, it should be more suitable to 'pause' on the capacity itself at azure side.

Pause and resume your capacity - Microsoft Fabric | Microsoft Learn

Update: thanks for frithjof_v 's reminder, I update and modify the reference link to notebookutils.

Regards,

Xiaoxin Sheng

Re: F2 Capacity. spark.stop() at the end of the notebook but still running after this command

frithjof_v — Mon, 12 Aug 2024 08:29:11 GMT

Anonymous Thanks for sharing,

I guess the best option is to have one "master" notebook orchestrate the running of the other notebooks, as you mention (I have never tried it myself).

However I think the most updated link for use in Fabric Notebook is this one: https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities

More specifically: https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#reference-a-notebook

This is for the Fabric Notebookutils.

The link you provided is for Azure Synapse Analytics. Or am I missing something?

I do however believe I have experienced that a function which was listed in Azure Synapse Analytics mssparkutils, but not listed in Fabric Notebookutils, still worked in Fabric Notebook.

But according to the docs, I think Fabric Notebookutils (ref. the links in this comment) should be the most updated one for use inside Fabric?