Service Principal account ran Data Pipeline

[sammydyson55]

In the Monitor tab I can see that our service principal account used to create our CI CD deployment pipeline in Dev Ops for some reason it ran the data pipeline twice once at 13/06/2025 4.39 pm and again 13/06/2025 at 4.40 pm 

How could this even happen, I have tried to replicate the problem but to no avail.

Pipeline run ID ed76ac16-1e00-488e-a31c-f024810bea6b and 65f1ce2b-0f5f-41b8-93a7-f30adbcdce03

Further to this when you drill through and look at the pipeline run detail it says my own account ran the pipeline which causes confusion as I did not run the pipeline twice at the same time

[sammydyson55]

Hi @v-tsaipranay and @burakkaragoz 

Thank you for your responses,

In my Dev Ops pipeline it is trying to move items between workspaces there is nothing in the Dev Ops pipeline that is running Data Pipelines in Fabric, I wasnt even aware this was even possible?

I tried replicating what I was doing at the time it occurred and I was unable to replicate the issue. 

The previous comment mentioned investigating through the monitoring unfortunately this was not setup however I have set this up now.

One thing I did notice is that when I run our master pipeline it runs under my user name however some of the child pipelines it uses the service principal account so would that be why it shows as the service principal account running it? This is a bit confusing as ultimately I was the user who ran the pipeline so it should be my user account that also ran the child pipelines?

Has it been reported elsewhere where users have experienced pipelines running twice at the same time ? This is the most concerning issue to me as we wont be able to use it for production purposes

[v-tsaipranay · Accepted solution]

Hi @sammydyson55 ,

Your DevOps pipeline primarily transfers items between workspaces and does not execute Fabric Data Pipelines. Therefore, it is unlikely that DevOps caused those pipeline runs. Triggering Fabric pipelines from DevOps via REST APIs requires a specific configuration; if that setup hasn't been implemented, we can exclude it as a cause.

Regarding the identity confusion, this behavior is expected with Fabric. If you manually trigger the master pipeline, it will show your name. However, if the child pipelines use connections running under a service principal, those runs will display the service principal as the identity. Fabric tracks who runs what at each step, so different identities between parent and child pipelines are normal depending on the setup.

About the two runs happening so close together that’s a valid concern. It’s not something we’ve seen reported often, but it can happen if there’s retry logic, overlapping triggers, or an external call that ran twice. Since monitoring wasn’t turned on at the time, it’s hard to know exactly what caused it in this case.

Now that you’ve enabled monitoring, that should give much more visibility if this happens again. If it does, reviewing the run logs should help pinpoint the cause.

I hope this suggestion helps you get a better idea. Please reach out for further assistance.

If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.

[burakkaragoz]

Hi @sammydyson55 ,

You've raised a great question, and the support answer above covers the main technical reasons really well. To add a bit more context and practical steps:

• In CI/CD environments with service principal automation, it's not uncommon for a pipeline to execute twice within a short window, especially if there’s a retry policy, accidental duplication in the YAML or pipeline configuration, or a transient failure that causes an automatic re-run.
• Regarding the user identity confusion: Sometimes, Fabric (or Azure) logs will display the user who initially created, last modified, or authenticated the pipeline, not necessarily the identity that actually triggered the run at execution time. This can happen if the service principal is delegated or if the metadata picks up the original pipeline author. So, even though the service principal triggered the job, your name might appear in the logs.

Here are a few additional tips:

1. Double-check your DevOps pipeline YAML and any trigger/retry logic for possible duplicate steps or settings that could have caused the double execution.
2. In the Fabric Monitor, for both run IDs, go to the detailed logs and look for:
   • What event or trigger started each run (was it a retry, a schedule, or a manual trigger?)
   • The exact parameters and context for each execution.
   • Any error, timeout, or retry message just before the second run started.
3. If audit logging is enabled, check the Activity Logs/Audit Logs to see which identity actually triggered the pipeline and whether there are any clues about delegated authentication.
4. If this behavior is not intended and you want to prevent duplicate runs, consider adding checks in your pipeline to block re-entrancy, or review conditions/triggers to ensure the task can’t be queued twice under normal circumstances.

Let us know what you find in the logs, or if you need help interpreting any of the pipeline details. Sometimes, sharing a snippet from the trigger section or the logs can help the community spot the root cause more quickly.

Hope this helps you get to the bottom of it!

[v-tsaipranay]

Hi @sammydyson55 

Thank you for reaching out to the Microsoft Fabric Community and for sharing the detailed observations regarding your pipeline executions.

Based on the information you've provided, it appears that the pipeline may have been triggered twice by your Azure DevOps setup, possibly due to how the deployment steps are configured. In CI/CD scenarios, especially when using service principals for automation, it's not unusual for a pipeline to run more than once if the task is accidentally duplicated in the YAML file or if there’s retry logic that causes it to trigger again. Since the two runs happened within a minute of each other, we recommend checking your DevOps run logs from that timeframe to see if a retry or repeat execution was triggered due to a timeout or an unhandled failure.

As for the confusion around the user identity shown in the run logs, where it shows your personal account instead of the service principal this could be due to how Microsoft Fabric captures and displays run metadata. In some cases, the execution context might reflect the user who originally configured or authenticated the pipeline, or it could be tied to a delegated token used by the service principal. This might explain why your name appears, even though the actual trigger came from the DevOps pipeline using the service principal.

We suggest reviewing the run details for both pipeline executions (ed76ac16-1e00-488e-a31c-f024810bea6b and 65f1ce2b-0f5f-41b8-93a7-f30adbcdce03) from the Fabric Monitor tab. Look closely at the trigger type, parameters, and any metadata that could indicate how and why each run occurred. If audit logging is enabled in your environment, checking the Activity Logs can also help identify exactly who or what triggered the pipeline and from where.

For your reference, here are some Microsoft Fabric documents that may help with further investigation:

How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn

Service principal support in Data Factory - Microsoft Fabric | Microsoft Learn

Automate Git integration by using APIs - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.