topic Re: Spark job taking long time to run in Data Engineering

Spark job taking long time to run

Kar_c — Wed, 05 Mar 2025 22:22:08 GMT

Hi, I have been using notebook to execute my spark jobs, I have noticed that it takes around a minute and half to complete the job in a pipeline - the data volume is really low and I am using F4 capacity.

When I explored more on the monitoring section I noticed that the time actually it(spark tasks) takes is around 4 secs but not sure why the job runs for 1.5 minutes in a pipeline.

I have a requirement of running the pipeline once in every 5 mins, so looking to optimise it

Need suggestions to handle this.

My pipeline:

Spark job - spark job(generic for the project) - copy job

Want the entire pipeline to complete within a minute, now it takes about 3 mins to complete. Currently the data volume is around 3000 rows for 1 run.

Re: Spark job taking long time to run

lbendlin — Thu, 06 Mar 2025 01:07:15 GMT

you are sitting nicely between a rock and a hard place. Spark pools take time to spin up. You could keep them "warm" in between runs but that means you will pay for that "keeping warm".

Five minutes is exactly the sweet/sour spot where you lose no matter how you play that. Try to run your jobs less frequently, or accept the additional cost.

Re: Spark job taking long time to run

v-prasare — Mon, 10 Mar 2025 21:48:30 GMT

@Kar_c, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Spark job taking long time to run

Kar_c — Mon, 10 Mar 2025 22:53:11 GMT

Thanks, we have not made any decisions as of now. I will further get intouch if I need more details.

Thanks again for your reply.

Re: Spark job taking long time to run

v-prasare — Mon, 17 Mar 2025 09:27:45 GMT

@Kar_c,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Spark job taking long time to run

v-prasare — Thu, 20 Mar 2025 08:29:44 GMT

@Kar_c, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query