topic Re: Incremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory in Pipelines

Incremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory

achintamondal21 — Wed, 22 Oct 2025 11:32:51 GMT

I have an NFS drive containing multiple levels of nested folders. I intend to identify the most recently modified files across all directories recursively and copy only last modifed files into a LakeHouse. I am seeking guidance on the recommended approach to implement this file copy operation using Microsoft Fabric Data Factory. An example of a source file path is:

1. \\XXX.XX.XXX.XXX\PROTOCOLS\ACTVAL\1643366695194009_SGM-3\221499200020\__NOPROGRAM___10004457\20240202.HTM
2. \\XXX.XX.XXX.XXX\PROTOCOLS\ACTVAL\1643366695194009_SGM-3\221499810020\__NOPROGRAM___10003395\20240202.HTM
3. \\XXX.XX.XXX.XXX\PROTOCOLS\ACTVAL\1760427099988857_P902\__NOORDER__\__NOPROGRAM_____NOMOLD__\20251014.HTM

Re: Incremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory

tayloramy — Wed, 22 Oct 2025 13:51:10 GMT

Hi achintamondal21,

You’re on the right track. What you want is a recursive file copy that only moves files changed since the last run. In Fabric Data Factory you can do this without hand-rolling recursion by using the File System connector (through the on-premises gateway), the Copy activity’s Recursively option, and Filter by last modified (i.e., a “watermark” window).

Below are two workable patterns:

Quick Solution (Incremental by last modified): copy all files whose LastModified is newer than your saved watermark (or the current run’s time window).
Alternative (Only the single latest file): enumerate files, compute the max LastModified, and copy just that file.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

Re: Incremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory

v-kpoloju-msft — Thu, 23 Oct 2025 06:54:51 GMT

Hi achintamondal21,

Thanks for reaching out to the Microsoft Fabric Community Forum. Also, thanks to tayloramy, for his inputs on this thread.

I understand you want to copy only the most recently modified files from your nested NFS folders into a Fabric Lakehouse using Data Factory. Since Fabric doesn’t have a native NFS connector, you’ll need to access your NFS via an On-premises Data Gateway or a network share that the pipeline can reach.

You can then use a Get Metadata activity with the “Recursive” option to list all files along with their Last Modified timestamps. Filter these files based on the last successful pipeline run (stored in a variable or Lakehouse table) to select only newly modified files and pass the filtered list to a Copy Data activity to move them into your Lakehouse.

Finally, maintain an incremental log of the latest processed timestamp in a control table or dataset so that future runs only ingest new or updated files. This approach ensures efficient incremental ingestion without duplicate copies.

For reference, here are relevant docs:
1. https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-onprem
2. https://learn.microsoft.com/en-us/fabric/data-factory/get-metadata-activity
3. https://learn.microsoft.com/en-us/fabric/data-factory/connector-lakehouse-copy-activity

Hope that clarifies. Let us know if you have any doubts regarding this. We will be happy to help.

Thank you for using the Microsoft Fabric Community Forum.

Re: Incremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory

v-kpoloju-msft — Sun, 26 Oct 2025 13:23:04 GMT

Hi achintamondal21,

Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.

Thank you.

Re: Incremental File Ingestion from NFS to LakeHouse using Microsoft Fabric Data Factory

v-kpoloju-msft — Wed, 29 Oct 2025 09:00:01 GMT

Hi achintamondal21,

Just wanted to follow up. If the shared guidance worked for you, that’s wonderful hopefully it also helps others looking for similar answers. If there’s anything else you'd like to explore or clarify, don’t hesitate to reach out.

Thank you.