Forum Discussion
Spark job taking long time to run
Hi, I have been using notebook to execute my spark jobs, I have noticed that it takes around a minute and half to complete the job in a pipeline - the data volume is really low and I am using F4 capacity.
When I explored more on the monitoring section I noticed that the time actually it(spark tasks) takes is around 4 secs but not sure why the job runs for 1.5 minutes in a pipeline.
I have a requirement of running the pipeline once in every 5 mins, so looking to optimise it
Need suggestions to handle this.
My pipeline:
Spark job - spark job(generic for the project) - copy job
Want the entire pipeline to complete within a minute, now it takes about 3 mins to complete. Currently the data volume is around 3000 rows for 1 run.
you are sitting nicely between a rock and a hard place. Spark pools take time to spin up. You could keep them "warm" in between runs but that means you will pay for that "keeping warm".
Five minutes is exactly the sweet/sour spot where you lose no matter how you play that. Try to run your jobs less frequently, or accept the additional cost.
5 Replies
- lbendlin
Super User
you are sitting nicely between a rock and a hard place. Spark pools take time to spin up. You could keep them "warm" in between runs but that means you will pay for that "keeping warm".
Five minutes is exactly the sweet/sour spot where you lose no matter how you play that. Try to run your jobs less frequently, or accept the additional cost.
Thanks, we have not made any decisions as of now. I will further get intouch if I need more details.
Thanks again for your reply.
v-prasareCommunity Support
Kar_c, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query
- v-prasare
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query
- v-prasare
Kar_c, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query
