If the Data Pipeline has last been saved (changed or modified) by a Service Principal or a Managed Identity, and the Data Pipeline has a Spark Job Definition step, then the execution step fails with the 'Forbidden' http status code.
Error code: 2474
Failure type: User configuration issue
Details: "Spark job definition execution failed to start with http status code - 'Forbidden', additional details - 'System.Net.Http.StreamContent' : "
Both the Data Pipeline and Spark Job Definition are items in the same Workspace. The Service Principal (or Managed Identity) is owner on both items and it has the all of the permissions on the Spark Job Definition (Read, Write, Reshare, Execute, ReadAll, ViewOutput, ViewLogs).
We have observed, if one user is the last one to save the Data Pipeline, then when another user is starting the run, the submitter of the Spark Job Definition execution run is then the user, who last saved the Data Pipeline. Meaning, it is not the user who started the Data Pipeline execution run there ended up being the submitter.
It seams like, that the Data Pipeline are using the wrong credentials to access the Spark Job Definition item.
Another thing we have observed is, that if a Spark Job Definition has been manually triggered by a Service Principal (or a Managed Identity), then the submitter area is empty.
The only solution we can think about right now is not to use the Data Pipeline to execute Spark Job Definition simultaneously or one at a time, but making a script to do this for us. Anybody, with a better idea?
