Forum Discussion

VIvMouret's avatar
VIvMouret
Frequent Visitor
1 year ago
Solved

Spark Job Definition is still restarting himself

Hello everyone,


I've converted one of my notebooks into an Spark Job Definition (SJD).

I had no trouble understanding how everything worked, I like it.

I just have one "problem" (if I can call it a problem?)

I'm still in the development phase, so breakages are possible,

I saw the optimisation option in the settings that lets you restart the SJD x times (with or without an interval, etc).


Except that I prefer to disable it, so logically my SJD launches, fails at a specific point and stops.
I've just noticed that it very often - if not always - relaunches just once, as if it were trying to catch up.


I don't understand, I've disabled optimisation, it should fail and stop there, so why does it try again? Not that it's impractical in production, but in the test phase it's odd to see this behaviour, which is supposed to be deactivated

 

Thanks for the info

 

 

EDIT: 

I've just had the same problem again today. You can see in this screenshot the send time and the start time.

A run went through, failed but the SJD decided to relaunch for X reasons

Spark Job Definition

6 Replies

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Icon for Community Support rankCommunity Support
    1 year ago

    Hi VIvMouret,

    Thank you for reaching out to the Microsoft Fabric Forum Community. 

     

    The “Retry on failure” option is disabled under the Optimization tab of the SJD settings.

    Please ensure that the retry on failure is disabled and set the retry count to 0.  

    After a restart, check the job run history and look for early-stage failure logs to determine if the retry was due to infrastructure issues.

     

    If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

     

    Thank you. 

    • VIvMouret's avatar
      VIvMouret
      Frequent Visitor
      1 year ago

      Yes the option is well disabled

      As you can see, I can't set the retry count at 0. Even if I write 0, it will write 1 just after.
      I can't get any logs as it reboots and resets the logs.

       

      I'm not sure if you understood correctly :

      - I have no history of the reset run

      - the option is disabled

      And in spite of everything, at the first error it loops again and stops if it receives an error