Forum Discussion
Error/warnings during Delta table write in Spark Fabric Notebooks
Hi aa_tsl,
Thank you for reaching out in Microsoft Community Forum.
The long execution time is likely due to Spark job overhead in Fabric's F2 SKU, especially when using .collect() on small datasets. Even simple operations can feel slow because of cluster orchestration time, not computation itself.
Please follow below steps to resolve the issue;
1.Replace .collect() with .toPandas() for small datasets to reduce overhead.
2.Keep transformations within Spark as long as possible (avoid switching to Python lists too early).
3.For <2000 rows, consider using Pandas instead of Spark — it’s faster for small data.
4.Fabric's F2 SKU has higher latency for small jobs; if possible, test on F4 for better responsiveness.
Please continue using Microsoft Community Forum.
If this post helpes in resolve your issue, kindly consider marking it as "Accept as Solution" and give it a 'Kudos' to help others find it more easily.
Regards,
Pavan.
Hi aa_tsl,
Thank you for reaching out in Microsoft Community Forum.
Please follow below steps to resolve the issue;
1.Avoid using .distinct() on small datasets as it triggers expensive shuffling. Use .dropDuplicates() if deduplication is needed.
2.Do not use partitionBy for small datasets; let Spark handle partitioning automatically to reduce overhead.
3.Set spark.sql.shuffle.partitions to a lower value (e.g., 😎 and use spark.sql.files.maxPartitionBytes to control partition sizes.
4.Ensure sufficient resources (memory, CPU) in the Fabric F2 SKU cluster and set the log level to DEBUG for further insights into the issue.
Please continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.