Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
MsMartin
Regular Visitor

Issues when manual triggering a Data Pipeline run with a Spark Job Definition step

If the Data Pipeline has last been saved (changed or modified) by a Service Principal or a Managed Identity, and the Data Pipeline has a Spark Job Definition step, then the execution step fails with the 'Forbidden' http status code.

Error code: 2474
Failure type: User configuration issue
Details: "Spark job definition execution failed to start with http status code - 'Forbidden', additional details - 'System.Net.Http.StreamContent' : "

Both the Data Pipeline and Spark Job Definition are items in the same Workspace. The Service Principal (or Managed Identity) is owner on both items and it has the all of the permissions on the Spark Job Definition (Read, Write, Reshare, Execute, ReadAll, ViewOutput, ViewLogs).

We have observed, if one user is the last one to save the Data Pipeline, then when another user is starting the run, the submitter of the Spark Job Definition execution run is then the user, who last saved the Data Pipeline. Meaning, it is not the user who started the Data Pipeline execution run there ended up being the submitter.

It seams like, that the Data Pipeline are using the wrong credentials to access the Spark Job Definition item.

Another thing we have observed is, that if a Spark Job Definition has been manually triggered by a Service Principal (or a Managed Identity), then the submitter area is empty.

 

The only solution we can think about right now is not to use the Data Pipeline to execute Spark Job Definition simultaneously or one at a time, but making a script to do this for us. Anybody, with a better idea? 

1 ACCEPTED SOLUTION
v-prasare
Community Support
Community Support

Hi @MsMartin , 

If your getting blocked, In this scenario i suggest you to raise a support ticket here. so, that they can assit you in addressing the issue you are facing. please follow below link on how to raise a support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

 

 

I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. Feedback submitted here is often reviewed by the product teams and can lead ...

 

 

Thanks,

Prashanth Are

View solution in original post

3 REPLIES 3
v-prasare
Community Support
Community Support

Hi, We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

 

Thank you for your understanding and participation.

Hi. I have made a support ticket for this, but it is not resolved yet. I don't even think, there have been made a bug for it yet. Im in dialog with the people taking care of the ticket.
Thank you.

v-prasare
Community Support
Community Support

Hi @MsMartin , 

If your getting blocked, In this scenario i suggest you to raise a support ticket here. so, that they can assit you in addressing the issue you are facing. please follow below link on how to raise a support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

 

 

I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. Feedback submitted here is often reviewed by the product teams and can lead ...

 

 

Thanks,

Prashanth Are

Helpful resources

Announcements
July 2025 community update carousel

Fabric Community Update - July 2025

Find out what's new and trending in the Fabric community.

June FBC25 Carousel

Fabric Monthly Update - June 2025

Check out the June 2025 Fabric update to learn about new features.

Top Solution Authors