topic Re: Issues when manual triggering a Data Pipeline run with a Spark Job Definition step in Pipelines

Issues when manual triggering a Data Pipeline run with a Spark Job Definition step

MsMartin — Mon, 28 Apr 2025 18:25:30 GMT

If the Data Pipeline has last been saved (changed or modified) by a Service Principal or a Managed Identity, and the Data Pipeline has a Spark Job Definition step, then the execution step fails with the 'Forbidden' http status code.

Error code: 2474
Failure type: User configuration issue
Details: "Spark job definition execution failed to start with http status code - 'Forbidden', additional details - 'System.Net.Http.StreamContent' : "

Both the Data Pipeline and Spark Job Definition are items in the same Workspace. The Service Principal (or Managed Identity) is owner on both items and it has the all of the permissions on the Spark Job Definition (Read, Write, Reshare, Execute, ReadAll, ViewOutput, ViewLogs).

We have observed, if one user is the last one to save the Data Pipeline, then when another user is starting the run, the submitter of the Spark Job Definition execution run is then the user, who last saved the Data Pipeline. Meaning, it is not the user who started the Data Pipeline execution run there ended up being the submitter.

It seams like, that the Data Pipeline are using the wrong credentials to access the Spark Job Definition item.

Another thing we have observed is, that if a Spark Job Definition has been manually triggered by a Service Principal (or a Managed Identity), then the submitter area is empty.

The only solution we can think about right now is not to use the Data Pipeline to execute Spark Job Definition simultaneously or one at a time, but making a script to do this for us. Anybody, with a better idea?

Re: Issues when manual triggering a Data Pipeline run with a Spark Job Definition step

v-prasare — Tue, 29 Apr 2025 09:29:01 GMT

Hi MsMartin ,

If your getting blocked, In this scenario i suggest you to raise a support ticket here. so, that they can assit you in addressing the issue you are facing. please follow below link on how to raise a support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

Thanks,

Prashanth Are

Re: Issues when manual triggering a Data Pipeline run with a Spark Job Definition step

v-prasare — Fri, 30 May 2025 04:50:32 GMT

Hi, We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

Re: Issues when manual triggering a Data Pipeline run with a Spark Job Definition step

MsMartin — Mon, 02 Jun 2025 08:23:40 GMT

Hi. I have made a support ticket for this, but it is not resolved yet. I don't even think, there have been made a bug for it yet. Im in dialog with the people taking care of the ticket.
Thank you.