Forum Discussion
Error/warnings during Delta table write in Spark Fabric Notebooks
Hi aa_tsl,
Thank you for reaching out in Microsoft Community Forum.
The long execution time is likely due to Spark job overhead in Fabric's F2 SKU, especially when using .collect() on small datasets. Even simple operations can feel slow because of cluster orchestration time, not computation itself.
Please follow below steps to resolve the issue;
1.Replace .collect() with .toPandas() for small datasets to reduce overhead.
2.Keep transformations within Spark as long as possible (avoid switching to Python lists too early).
3.For <2000 rows, consider using Pandas instead of Spark — it’s faster for small data.
4.Fabric's F2 SKU has higher latency for small jobs; if possible, test on F4 for better responsiveness.
Please continue using Microsoft Community Forum.
If this post helpes in resolve your issue, kindly consider marking it as "Accept as Solution" and give it a 'Kudos' to help others find it more easily.
Regards,
Pavan.
Hi, I'm facing the same issue while reading a csv file containing 10K rows. It takes more than a minute to read the file and the logs contain following error (1st attempt fails after 55 seconds with below error and the 2nd attempt succeeds in less than a second). Strangely when I change the extension of the file to .txt and read it as a text file, it doesn't give below error and executes in 2 seconds. Also we are using Large sized nodes and the size of the file is 500KB, so I doubt if the executor size is an issue. Any other potential rootcauses here?
2025-08-27 17:59:45,913 ERROR TaskResources [Executor task launch worker for task 0.0 in stage 1.0 (TID 1)]: Task 1 failed by error: java.io.IOException: Pipe has no content; awaitReadable() returned false [/mnt/vegas/pipes/75cd6630-ad17-4ad7-83d9-b9decdbaa50b.pipe, pos=0,blocksRead=0; bytesRead=0; availInPipe=0] Vegas Service: Context=f8114d46-8034-4bcc-b04e-23773df9339f, Abandoned pipe - unread by the time it was closed at org.apache.hadoop.shaded.com.microsoft.vegas.common.VegasPipeInputStream.pipelineThread(VegasPipeInputStream.java:536) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)