topic Spark Job Definition is still restarting himself in Data Engineering

Spark Job Definition is still restarting himself

VIvMouret — Mon, 19 May 2025 14:42:50 GMT

Hello everyone,

I've converted one of my notebooks into an Spark Job Definition (SJD).

I had no trouble understanding how everything worked, I like it.

I just have one "problem" (if I can call it a problem?)

I'm still in the development phase, so breakages are possible,

I saw the optimisation option in the settings that lets you restart the SJD x times (with or without an interval, etc).

Except that I prefer to disable it, so logically my SJD launches, fails at a specific point and stops.
I've just noticed that it very often - if not always - relaunches just once, as if it were trying to catch up.

I don't understand, I've disabled optimisation, it should fail and stop there, so why does it try again? Not that it's impractical in production, but in the test phase it's odd to see this behaviour, which is supposed to be deactivated

Thanks for the info

EDIT:

I've just had the same problem again today. You can see in this screenshot the send time and the start time.

A run went through, failed but the SJD decided to relaunch for X reasons

Re: Spark Job Definition is still restarting himself

v-saisrao-msft — Mon, 19 May 2025 06:14:48 GMT

Hi VIvMouret,

Thank you for reaching out to the Microsoft Fabric Forum Community.

The “Retry on failure” option is disabled under the Optimization tab of the SJD settings.

Please ensure that the retry on failure is disabled and set the retry count to 0.

After a restart, check the job run history and look for early-stage failure logs to determine if the retry was due to infrastructure issues.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

Re: Spark Job Definition is still restarting himself

VIvMouret — Mon, 19 May 2025 08:19:26 GMT

Yes the option is well disabled

As you can see, I can't set the retry count at 0. Even if I write 0, it will write 1 just after.
I can't get any logs as it reboots and resets the logs.

I'm not sure if you understood correctly :

- I have no history of the reset run

- the option is disabled

And in spite of everything, at the first error it loops again and stops if it receives an error

Re: Spark Job Definition is still restarting himself

v-saisrao-msft — Tue, 20 May 2025 07:32:28 GMT

Hi VIvMouret,

Thank you again for your detailed explanation, and sorry for the inconvenience caused.

I suggest you to raise a support ticket here. so, that they can assit you in addressing the issue you are facing. please follow below link on how to raise a support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn.

Thank you.

Re: Spark Job Definition is still restarting himself

VIvMouret — Mon, 26 May 2025 13:55:32 GMT

Hello v-saisrao-msft,

Thanks for the tip, I've just done it!

Hope it helps!

Best regards

Re: Spark Job Definition is still restarting himself

v-saisrao-msft — Tue, 27 May 2025 10:35:35 GMT

Hi VIvMouret,

Thank you for the update.

Re: Spark Job Definition is still restarting himself

v-saisrao-msft — Thu, 29 May 2025 13:03:48 GMT

Hi VIvMouret,

Could you please confirm if the issue was resolved after raising a support case? If so, we’d appreciate it if you could share the solution to help others in the community.

As we haven’t heard back, we’re closing this thread per our follow-up policy. For any further issues, please start a new thread in the Microsoft Fabric Community Forum we’ll be happy to assist.

Thank you for being part of the Microsoft Fabric Community.