topic Re: Need suggestions to run our production data pipelines in Data Engineering

Need suggestions to run our production data pipelines

ati_puri — Tue, 03 Jun 2025 07:09:47 GMT

In our Dev environment, we currently use user-based authentication to run data pipelines in Fabric for Bronze, Silver and Gold. However, for the production environment, we are exploring best practices that do not involve service accounts. Can someone advise the best method/or available options to securely run data pipelines in production?

Re: Need suggestions to run our production data pipelines

Anonymous — Tue, 03 Jun 2025 10:36:12 GMT

Hi ati_puri ,
Thank you for reaching out to us on Microsoft Fabric Community Forum!

To securely run pipelines in production without user credentials or traditional service accounts, Microsoft recommends using Workspace Identity. Please refer the below links for more information.
https://learn.microsoft.com/en-us/fabric/security/workspace-identity-authenticate

Also refer the solved thread which might help you :
https://community.fabric.microsoft.com/t5/Data-Engineering/Connecting-to-Fabric-database-using-workspace-identity/m-p/4414936

Hope this answer help you to resolve the query.If so,give us kudos and consider accepting it as solution.

Regards,
Pallavi G.

Re: Need suggestions to run our production data pipelines

BhavinVyas3003 — Tue, 03 Jun 2025 10:42:25 GMT

Hi ati_puri

Use Managed Identity instead of user or service accounts.

Enable Managed Identity in your production Fabric workspace (via Admin portal).
Grant access to required data sources (e.g., Azure Data Lake, SQL) using RBAC or ACLs.
Configure pipelines to use the workspace’s Managed Identity for authentication.

Re: Need suggestions to run our production data pipelines

Ilgar_Zarbali — Wed, 04 Jun 2025 09:49:21 GMT

For production pipelines in Fabric, the best practice is to use managed identities or workspace-managed credentials instead of user-based authentication. This avoids dependency on personal accounts or service accounts. Here are your secure options:

Workspace-managed identity: Fabric can use a system-assigned identity to authenticate securely with data sources (like Azure SQL, Data Lake, etc.).
Linked services with credential store: Set up linked services using secure credential management and reuse them in pipelines and dataflows.
Access control via Azure AD: Ensure all resources grant access to the Fabric workspace identity, not individual users.

This setup improves security, avoids credential expiration issues, and supports automation in production.

Re: Need suggestions to run our production data pipelines

ati_puri — Wed, 04 Jun 2025 10:17:54 GMT

Can we use WS Identity in Azure Devops to trigger our data pipelines schedules or triggers?

Re: Need suggestions to run our production data pipelines

ati_puri — Fri, 06 Jun 2025 14:15:29 GMT

Managed Identity is not supported with Azure Devops to trigger data pipelines