topic Re: Setting a Spark cluste when running a notebook or running anything else(eg. pipeline, dataflow etc)? in Pipelines

Setting a Spark cluste when running a notebook or running anything else(eg. pipeline, dataflow etc)?

Anonymous — Tue, 22 Oct 2024 01:43:23 GMT

Normally in Databricks I can set the cluster that a notebook would run on in the box in the top right of a notebook. However in a Fabric notebook, I can't see it in the top right corner anymore.

How can I assign a cluster to a notebook or any other activity eg. pipeline?

Re: Setting a Spark cluste when running a notebook or running anything else(eg. pipeline, dataflow etc)?

frithjof_v — Tue, 22 Oct 2024 07:15:31 GMT

For Notebook, it's in the Environment dropdown.

For general use cases, I think I would just use the default, i.e. Starter pool, as they have shorter startup times.

In that case, you don't need to think about cluster. The Notebook will automatically use the default pool (cluster) if we don't specify otherwise.

I think only Notebooks (and Spark Job Definitons) use Spark.

I'm not aware that Data Pipeline or Dataflow Gen2 use Spark. I think they use another technology which is fully managed (hidden from us).

If you run a Notebook in Data Pipeline, it will use Spark.

Here is a link to some information about Apache Spark in Fabric:

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

Re: Setting a Spark cluste when running a notebook or running anything else(eg. pipeline, dataflow etc)?

Srisakthi — Tue, 22 Oct 2024 06:45:01 GMT

Hi Anonymous ,

You can set the spark properties in Workspace setting, you can create pool based on your requirements for spark nodes and attach to your workspace. You can create new environment on which you can specify the executor core, memory, dynamic allocation to be used. using Spark properties tab you can add your environment specific spark properties, Libraries.

please refer screenshot -

Thanks,

Srisakthi

---------------------------------------------------------------------------

If this answers your question please mark as solution accepted.