Error/warnings during Delta table write in Spark Fabric Notebooks

Hi aa_tsl,

Thank you for reaching out in Microsoft Community Forum.

The long execution time is likely due to Spark job overhead in Fabric's F2 SKU, especially when using .collect() on small datasets. Even simple operations can feel slow because of cluster orchestration time, not computation itself.

Please follow below steps to resolve the issue;

1.Replace .collect() with .toPandas() for small datasets to reduce overhead.

2.Keep transformations within Spark as long as possible (avoid switching to Python lists too early).

3.For <2000 rows, consider using Pandas instead of Spark — it’s faster for small data.

4.Fabric's F2 SKU has higher latency for small jobs; if possible, test on F4 for better responsiveness.

Please continue using Microsoft Community Forum.

If this post helpes in  resolve your issue, kindly consider marking it as "Accept as Solution" and give it a 'Kudos' to help others find it more easily.

Regards,
Pavan.