Forum Discussion

Marco117's avatar
Marco117
Frequent Visitor
1 year ago

Ms Fabric spark notebook with poor performance

I have a notebook that was running on a Databricks cluster with only 4 executor cores and 4 driver cores, and Autoscale and Dynamically allocate executors were disabled. Here, the notebook executed i...
data science
environment
notebook
prasbharat's avatar
prasbharat
Frequent Visitor
1 year ago

Marco117 

It sounds like you’re experiencing significant differences in execution time when running your Spark notebook in Microsoft Fabric compared to Databricks. Based on your description, here are some additional suggestions that could help optimize performance without conflicting with the previous insights provided:

1. Validate Resource Allocation and Execution Overhead

Use the Monitor Run series to check where the additional time is being spent. Running the notebook directly might yield insights into whether the pipeline execution is introducing delays.

  • If there are multiple notebooks in the pipeline, consider enabling High Concurrency Mode to minimize sequential bottlenecks.
  • If concurrency doesn’t apply, you can try running the notebooks independently for a direct comparison.

2. Since Autoscale and Dynamic Allocation are enabled, they might be overprovisioning resources unnecessarily. If your workload is stable and doesn’t require frequent scaling:

  • Disable Autoscale and Dynamic Allocation temporarily.
  • Configure fixed resources for Spark using settings similar to your Databricks cluster:

spark.conf.set("spark.executor.instances", "2")
spark.conf.set("spark.executor.cores", "4")
spark.conf.set("spark.driver.cores", "4")
spark.conf.set("spark.sql.shuffle.partitions", "8")

 

3. Ensure your input data is stored in an optimized format like Parquet or Delta for better read performance. If the current format is CSV or JSON, consider converting it:

df = spark.read.csv("path_to_data").repartition(4).write.parquet("optimized_data_path")

Partition your dataset based on the number of executor cores to reduce shuffle overhead:

 

df = df.repartition(4)

 

4. Monitor Job performance:

 

  • Enable Spark UI in Fabric to analyze job execution. Look for long-running stages or inefficient shuffling during transformations.
  • Use the explain() method to analyze your query execution plans and optimize joins, filters, or other transformations: df.explain()

If this post helps you resolve the issue, please consider accepting it as the solution to help other members find it more quickly. If you still have additional questions, feel free to let me know, and I’ll be happy to assist further. Thanks a lot!