Spark in Notebook taking more time to process the data.

Hi NagaRK ,

If your Spark notebook is taking longer to process data in Microsoft Fabric, one effective way to optimize performance is by configuring the Spark environment with resource profiles.

In Microsoft Fabric, resource profiles are integrated directly into the Spark environment. You can apply them by setting the appropriate configuration before executing your workload:

spark.conf.set("spark.fabric.resourceProfile", "readHeavyForSpark")

This sets the Spark environment to use the readHeavyForSpark profile, which is optimized for read-intensive operations. Other available profiles include:

• readHeavyForPBI
• writeHeavy
• custom
  These profiles adjust Spark's internal resource allocation—like executor memory, cores, and task parallelism—based on workload type.

 Reference Documentation: Configure Resource Profile Configurations in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Additionally, consider increasing Spark pool capacity by scaling up nodes or adjusting pool size in Fabric settings.