From ADF Inventory to a Fabric Operating Model: A Practical Migration Playbook

[ssrithar]

Most Azure Data Factory (ADF) environments did not start out complex. A team adds a pipeline for one source, another for a file feed, then a few parameters and a control table. A year or two later, that estate may include nested pipelines, watermark logic, configuration-driven loops, custom audit tables, and dependencies that only a few people fully understand.

That is why an ADF-to-Fabric migration should not begin with a goal to recreate every pipeline. In practice, it is more useful to ask three questions:

What is this workload actually doing? Which Fabric pattern is the right fit? And how will we know it is safe to switch it over?

This article shares a practical approach for teams moving a metadata-driven ADF estate to Microsoft Fabric. It is based on the kinds of decisions that tend to matter in a real implementation: source connectivity, incremental loads, operational support, reconciliation, and the temptation to make every workload fit one tool. The examples are generalized implementation patterns, not a description of one customer environment.

Start with an inventory, but make it useful

An ADF inventory is often treated as a list of pipelines to migrate. That is a missed opportunity. The inventory should instead help you decide whether a workload should move now, move later, or be redesigned first.

For each workload, capture more than its name and schedule:

What to Capture	Why it Matters
Source and source type	A SQL database, SaaS application, REST API, SharePoint site, and on-premises file share have very different Fabric options.
Load style	Full loads, watermark loads, CDC, event-driven feeds, and near-real-time replication should not be treated as the same problem.
Transformation work	Simple copies are very different from Spark processing, complex SQL, API paging, or data-quality rules.
Orchestration needs	Dependencies, loops, notifications, retries, and downstream processing can be the real reason a pipeline exists.
Connectivity dependencies	Record any use of SHIR, VNet integration runtime, private endpoints, Key Vault references, or a non-standard authentication method.
Operational expectations	Note the SLA, restart approach, audit requirements, business criticality, and the person or team that supports it.

This changes the conversation from “How many pipelines have we converted?” to “Which workloads are genuinely ready?”

Microsoft also recommends an assessment-first, phased approach. The migration experience can help identify pipeline and activity readiness, but it does not remove the need to validate linked services, triggers, global parameters, integration runtimes, Key Vault references, and custom logic around the pipeline. Migration planning for ADF to Fabric

For a first migration wave, I would choose 10–20 low-risk, high-value workloads, plus one representative orchestration chain. A simple single-table copy is useful for testing connectivity, but it will not tell you whether your operating model works.

Choose the Fabric pattern based on the work, not the product name

Fabric gives us several good options for moving data. The important point is that they solve different problems.

If you need to…	Consider…	Why it fits
Replicate supported operational data continuously for analytics	Mirroring	It provides managed, CDC-based replication into OneLake with minimal custom ingestion code.
Move a simple batch or incremental dataset	Copy job	It keeps straightforward ingestion lightweight, with less orchestration to maintain.
Drive many loads from metadata and include dependencies, audit steps, or conditional logic	Pipeline + Copy activity	It gives you an orchestration layer around the copy: validate, load, transform, reconcile, and log the result.
Handle custom APIs, files that need custom parsing, complex transformations, or reusable quality checks	Notebook, orchestrated by a Pipeline	It is the right place for code when the work really calls for code.
Access data already governed in another OneLake location	OneLake shortcut	It can avoid another physical copy when live access is what you need.
Keep a workload running while a dependency is not ready or supported	Retain ADF for now	A phased migration is safer than a forced redesign against a deadline.

Fabric’s current data-movement guidance follows a similar distinction: Mirroring for continuous replication, Copy job for simpler ingestion, and Pipelines when the workflow requires more orchestration and control. Connector support and feature behavior change over time, so confirm the supported source and destination combinations before choosing a production pattern. Fabric data movement decision guide

One decision that regularly comes up is whether to use Mirroring for every database source. Mirroring is excellent when the goal is supported operational replication for analytics. It is not designed to replace the rest of a multi-step process: source-readiness checks, custom transformation steps, special audit requirements, or dependencies across systems. If those controls are central to the workload, a Pipeline-based pattern is usually a better fit.

Likewise, a notebook should not be the default answer to every difficult source. If the task is essentially a parameterized copy, adding code can make operations harder without adding value. Use notebooks where they provide a clear benefit: handling an API, processing files whose structure varies, applying complex transformations, or enforcing reusable data-quality rules.

Keep the configuration-driven approach, but simplify it

Many mature ADF implementations already have a folder-load or configuration-driven framework. That is worth preserving. In fact, it is usually the best way to avoid replacing one large collection of pipelines with another.

Rather than creating a pipeline for every source table, use a small set of reusable Fabric pipelines controlled by metadata. A basic control table might look like this:

Column	Example use
SourceSystem	Groups loads by application or business domain.
SourceType	Identifies SQL Server, Oracle, REST API, SharePoint, storage, and so on.
SourceObject	Stores the table, view, path, file pattern, or API endpoint.
TargetLakehouse and TargetTable	Defines where the data lands.
LoadPattern	Specifies Full, Watermark, CDC where supported, or another agreed ingestion pattern.
WatermarkColumn and LastSuccessfulWatermark	Controls incremental extraction and restart behavior.
LoadPriority	Helps sequence loads when ordering matters.
Enabled	Lets support teams safely turn a load on or off without changing code.
ValidationRule	Identifies the minimum reconciliation required before the load is marked successful.

The pipeline flow is then consistent, even when the source is not:

1. Read the enabled configuration rows for a source system or load group.

2. Select the correct route for each row: Copy activity, Copy job, or notebook.

3. Load the data and capture row counts and run times.

4. Run the configured validation.

5. Commit the new watermark only after the load and validation succeed.

6. Write the final status, error details, and recovery information to an audit table.

7. Start downstream processing only when the ingestion result is trustworthy.

The operational benefit is significant. Onboarding a standard table may become a metadata change instead of a new pipeline build. More importantly, audit logging, retry behavior, validation, and restart rules stop being implementation details that vary from pipeline to pipeline.

There are also deliberate platform differences to account for. In Fabric, linked services become Connections, dataset definitions are typically configured inline in activities, and global parameters map to Variable Libraries. ADF self-hosted integration runtime scenarios use an On-premises Data Gateway in Fabric. Workloads that depend on ADF VNet integration runtime, private networking, or existing secret-access patterns need their own design validation; do not assume that they transfer unchanged. Differences between Azure Data Factory and Fabric Data Factory Migration planning for ADF to Fabric

A successful run is not yet a successful migration

The first successful Fabric load is a useful milestone. It is not evidence that the migration is complete.

For each production workload, run ADF and Fabric in parallel for an agreed period and compare their outputs. The comparison should be based on the way the business trusts the data today—not simply on whether the activity returned Success.

Check	What good looks like
Row counts	The source, ADF target, and Fabric target agree within a documented threshold.
Watermark range	Fabric processed the same intended incremental range as ADF.
Schema	Required columns, types, nullability, and key fields are correct.
Business aggregates	Measures such as amounts, quantities, or balances reconcile.
Duplicate handling	Re-running a failed load does not introduce duplicate business keys.
Timeliness	The Fabric run meets the agreed SLA under realistic volume.
Failure recovery	Support can restart safely and understand what was processed.
Observability	The team can identify the failed step, object, error, and recovery action without digging through multiple tools.

One detail is easy to miss: do not advance the watermark just because the extract completed. If the load or validation fails after extraction, advancing it can leave an unprocessed gap. Commit the watermark only after the destination result is validated and the audit record shows success.

Before turning on the Fabric production schedule, make the cutover gate explicit: technical reconciliation, performance validation, security review, and business-owner sign-off should all be complete. If a result is outside the agreed threshold, ADF stays in production while the Fabric design is corrected. That is not a failed migration; it is exactly what a controlled parallel run is meant to reveal.

The migration mistakes worth avoiding

Here are the patterns I would be most careful about:

• Treating Mirroring as a replacement for an end-to-end workflow. It handles replication well; it does not replace every dependency or operational control.

• Rebuilding every ADF pipeline as-is. First retire duplicates, unused pipelines, and temporary workarounds that no longer need to exist.

• Leaving connectivity and security checks until the last week. SHIR, network isolation, gateway needs, and secret-access patterns can determine the migration sequence.

• Calling an initial load “production-ready.” The more revealing tests are late-arriving data, schema drift, partial failure, restart, and the next incremental run.

• Hardcoding table names, source paths, and rules inside the framework. Keep standard behavior in metadata, and reserve code for genuine exceptions.

A phased way to run the program

The following sequence has worked well as a practical operating model:

1. Assess: classify workloads, dependencies, risk, and business criticality.

2. Pilot: migrate a representative group, including at least one configuration-driven or nested orchestration scenario.

3. Standardize: establish shared connections, control tables, audit tables, naming conventions, and deployment practices.

4. Validate: run ADF and Fabric together and reconcile the data, SLA, recovery behavior, and operational visibility.

5. Cut over: move the schedule after approval, retain an agreed rollback path, and retire ADF only after a stabilization period.

                                                     

Final thought

The goal is not to prove that every ADF pipeline can be rebuilt in Fabric. The goal is to leave the team with a simpler and more supportable data platform: fewer one-off pipelines, clear routing decisions, standard controls, and evidence that the data is right.

Start with the workload rather than the tool. Use the simplest Fabric pattern that meets the operational need. Then let reconciliation—not optimism—decide when it is ready for production.

In the next article, I will go deeper into the reusable Fabric ingestion framework itself: control-table design, audit tables, watermark handling, pipeline flow, and where notebooks fit.