Forum Discussion
Replicating Move behavior in Fabric pipelines
Hi arpost
When we were implementing this sort of behaviour, I typically used mv operation in the Spark notebook.
The link on the library:
https://learn.microsoft.com/en-us/fabric/data-engineering/microsoft-spark-utilities
While I can't guarantee how it works under the hood, but the "mv" command, it seems, does exactly what you want.
Also we observed using notebooks is very convenient way for that sort of operations. While you are mentioning the Data Pipeline, this approach is slightly different. But anyway, we didn't have any problems with that. Also, due to coding essence of the notebook, we noticed, it is much easier to configure exception handling during this sort of operations.
Conclusion. If notebook way is good for you, I strongly suggest to try it.
Thanks for good question and proactive position. I hope it helps. Kudo (like) and making answer as a solution will help me and others to contribute and use that contribution more effectively.
BR, Yurri
Hello arpost
I’d recommend moving each file only once into a Lakehouse landing folder (for example via SFTP/FTP) and then treating that landing zone as immutable. After this initial move, we avoid any further physical file movement.
Incremental ingestion using Spark
From the landing area, I’d use Spark Structured Streaming with checkpointing (Auto Loader pattern) to ingest files into the Bronze layer. The checkpoint becomes the source of truth for which files have already been processed.
How this avoids reprocessing
Rather than moving or deleting files to signal progress, Spark uses the checkpoint to automatically skip files it has already ingested. This allows landing files to remain in place without any risk of duplicate processing.
To control storage growth, I’d apply retention policies on the landing folders to automatically delete files after a defined number of days. This provides clean-up and compliance without introducing archive moves or extra compute cost.
Why this is better than a move-based approach
Compared to copy‑and‑delete “moves”, this significantly reduces Fabric capacity and IO cost by eliminating repeated data duplication. State is tracked logically via checkpoints, not physically by moving files around.
Operational benefits
Overall, this keeps the architecture simpler, more reliable, and easier to scale. It reduces custom filesystem code, lowers operational risk, and aligns well with modern Lakehouse ingestion best practices.