Forum Discussion
Error/warnings during Delta table write in Spark Fabric Notebooks
Hi aa_tsl,
Thank you for reaching out in Microsoft Community Forum.
The long execution time is likely due to Spark job overhead in Fabric's F2 SKU, especially when using .collect() on small datasets. Even simple operations can feel slow because of cluster orchestration time, not computation itself.
Please follow below steps to resolve the issue;
1.Replace .collect() with .toPandas() for small datasets to reduce overhead.
2.Keep transformations within Spark as long as possible (avoid switching to Python lists too early).
3.For <2000 rows, consider using Pandas instead of Spark — it’s faster for small data.
4.Fabric's F2 SKU has higher latency for small jobs; if possible, test on F4 for better responsiveness.
Please continue using Microsoft Community Forum.
If this post helpes in resolve your issue, kindly consider marking it as "Accept as Solution" and give it a 'Kudos' to help others find it more easily.
Regards,
Pavan.
Hi Anonymous , thanks for the reply.
I tried removing the .distinct() from the code, but it didn't have much effect 😕
It takes a lot of time for simple operations like this:
vendors_df = agents_df.filter(col("parent_id") == vendor_parent_id).withColumn(
"role", lit("vendor")
)
vendors_ids = [
row.agent_id for row in vendors_df.select("agent_id").collect()
]
advancing_vendors_df = agents_df.filter(
col("parent_id") == advancing_vendor_parent_id
).withColumn("role", lit("advancing_vendor"))
advancing_vendors_ids = [
row.agent_id for row in advancing_vendors_df.select("agent_id").collect()
]It took 2 minutes to run for a <2k dataset in fabric. When I run it locally (docker container simulating a spark environment it takes seconds, much faster).
Would you recommend using Pandas for data manipulation instead? Or do you have other tips on how to make Spark more performant on Fabric?
Best,
Alex
Hi aa_tsl,
Thank you for reaching out in Microsoft Community Forum.
The long execution time is likely due to Spark job overhead in Fabric's F2 SKU, especially when using .collect() on small datasets. Even simple operations can feel slow because of cluster orchestration time, not computation itself.
Please follow below steps to resolve the issue;
1.Replace .collect() with .toPandas() for small datasets to reduce overhead.
2.Keep transformations within Spark as long as possible (avoid switching to Python lists too early).
3.For <2000 rows, consider using Pandas instead of Spark — it’s faster for small data.
4.Fabric's F2 SKU has higher latency for small jobs; if possible, test on F4 for better responsiveness.
Please continue using Microsoft Community Forum.
If this post helpes in resolve your issue, kindly consider marking it as "Accept as Solution" and give it a 'Kudos' to help others find it more easily.
Regards,
Pavan.