Spark job taking long time to run

[Kar_c]

Hi, I have been using notebook to execute my spark jobs, I have noticed that it takes around a minute and half to complete the job in a pipeline - the data volume is really low and I am using F4 capacity.

When I explored more on the monitoring section I noticed that the time actually it(spark tasks) takes is around 4 secs but not sure why the job runs for 1.5 minutes in a pipeline. 

I have a requirement of running the pipeline once in every 5 mins, so looking to optimise it 

Need suggestions to handle this.

My pipeline: 

Spark job - spark job(generic for the project) - copy job

Want the entire pipeline to complete within a minute, now it takes about 3 mins to complete. Currently the data volume is around 3000 rows for 1 run.

[v-prasare]

@Kar_c, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare]

@Kar_c, 

 As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare]

@Kar_c, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[lbendlin · Accepted solution]

you are sitting nicely between a rock and a hard place.  Spark pools take time to spin up. You could keep them "warm"  in between runs but that means you will pay for that "keeping warm".

Five minutes is exactly the sweet/sour spot where you lose no matter how you play that. Try to run your jobs less frequently, or accept the additional cost.

[Kar_c]

Thanks, we have not made any decisions as of now. I will further get intouch if I need more details.

Thanks again for your reply.