Ms Fabric spark notebook with poor performance

Hi Marco117 ,

1. I tested running notebooks in Pipeline and running notebooks directly, and it took a few seconds longer to use Pipeline than to run notebooks (my query is simple), and you can further confirm where the time is being spent in the Monitor Run series. 

Monitor Apache Spark run series
Not sure how many Notebooks are in the pipeline, if there are more than one, consider using high concurrency mode.

Introducing High Concurrency Mode for Notebooks in Pipelines for Fabric Spark

If no exceptions are found above, then it is time to move to the autoscale and dynamically allocated actuators.

If your workload is relatively stable and you don't need additional cores to speed up execution, you might consider disabling these two features.

2. why using 72 cores:
When you enable autoscale for Spark pools, jobs exuecute with their minimum node configuration. During runtime, scaling may occur. These requests go through the job admission control. Approved requests scale up to the maximum limits based on total available cores. Rejected requests don't affect active jobs; they continue to run with their current configuration until cores become available. 
Job admission in Apache Spark for Fabric

3. Other possible reasons:
If you change the default pool from Starter Pool to a Custom Spark pool you may see longer session start (~3 minutes).
Both session and command execution times have increased in the first time. (It only took 20 seconds before.).

Workspace administration settings in Microsoft Fabric

Best Regards,
Gao
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly --  How to provide sample data in the Power BI Forum