Production-scale ADF → Fabric migration: REST API approach vs Microsoft Migration Utility

[dataengineerv2]

I'm automating Azure Data Factory → Microsoft Fabric Data Factory migration using ARM export + Fabric REST APIs instead of the Microsoft migration UI.

Current approach:

• Parse ARM templates (pipelines, datasets, linked services, etc.) using a custom parser.

• Create Fabric resources via REST APIs.

• Use a Service Principal (SPN) for authentication.

For connections:

• SP creates the connection using POST /v1/connections.

• SP then grants my user Owner using the role assignment API.

• The connection becomes visible in Manage Connections and behaves like a manually created one.

The approach is working for my current testing, but before investing further I'd like some feedback.

Has anyone used a similar approach for large ADF environments (100+ pipelines)? Any scalability, dependency, or production concerns that I should be aware of, especially around linked services, datasets, or API limitations?

Would appreciate any insights from people who have done enterprise-scale ADF → Fabric migrations.

[Vinodh247 · Accepted solution]

Your approach is valid and aligns with how many teams are bypassing the UI for scale, but at prod level there are a few realities you need to factor in. The Fabric REST APIs are still evolving, so expect gaps (especially around complex linked services, credentials, managed identity mappings, and dataflows) and occasional breaking changes, which the microsoft migration utility quietly handles for you.

Dependency orchestration is the biggest risk: ADF ARM templates are loosely ordered, but Fabric APIs require strict sequencing (connections -> datasets -> pipelines -> triggers), so you will need a robust dependency graph and retry logic. At 100+ pipelines, throttling and eventual consistency issues will show up, so build in idempotency, backoff, and state tracking rather than fire-and-forget deployments. Security-wise, SPN-created connections can behave differently with credential scoping and gateway bindings, so validate runtime execution, not just creation. Also watch for feature parity gaps (for ex, custom activities, mapping data flows, parameterization patterns) because some will not translate cleanly. Bottom line: your approach is scalable and more controllable than the UI, but only if you treat it like a full migration framework (stateful, versioned, retry-safe) rather than a onetime script.

[v-abhinavmu]

Hi @dataengineerv2,

Thanks for reaching out to the Microsoft Fabric Community forum.

Your proposed architecture is well suited for enterprise-scale ADF to Fabric migrations because it is scalable, repeatable, and can be fully automated. By parsing ARM templates to extract pipelines, datasets, linked services, triggers, and other resources, you can build an intermediate representation of the ADF assets before provisioning the corresponding Fabric resources through the REST APIs using a Service Principal. This approach also makes it easier to incorporate custom transformation logic, such as mapping ADF activities to their Fabric equivalents, and integrate the migration process. However, there are some considerations to keep in mind. Since Fabric continues to evolve, you'll need to maintain the mapping logic over time, and not every ADF activity or connector has a direct equivalent in Fabric. Additionally, certain resources, such as Linked Services, Integration Runtimes, and some connectors, may still require manual intervention or custom handling.

Alternate solution that you can use to migrate:

1) Using Powershell: PowerShell Migration of Azure Data Factory and Synapse Pipelines to Fabric - Microsoft Fabric | Mic…

2) Using Built in tool: Upgrade your Azure Data Factory pipelines to Fabric - Azure Data Factory | Microsoft Learn

Please check this article on best practices for Migration: Migration Best Practices for Azure Data Factory to Fabric Data Factory - Microsoft Fabric | Microso…

I hope this helps. Please feel free to reach out if you have any further questions.
Thank you.