Forum Discussion
Get file created date
Hello everyone.
I have a ingestion pipeline to copy files from a file system to ADLS. The goal is to copy all the files that have been uploaded into the source folder within the last 24 hrs. I am using the last modified date parameter in ADF to filter the files.
The issue I am running into is that when a older file is copied into the source folder from a different folder, the modified date is not changing (for ex - if I copied a file that was created a week ago into the source folder, the modified date will be that of a week ago and creation date would be the current time). This file should be considered as a new file and should be copied to the ADLS location, but it is not hapening.
Is there a way to acheive the functionality I want through ADF? Can we filter files based on their CREATION date instead of MODIFIED date?
Hi valjas2208,
In ADF’s file-based connectors, the built-in file filters are based on Last Modified time, not Creation time. So when you drop an old file into your “source” folder and the OS preserves its original modified timestamp, ADF’s “modifiedDatetimeStart/End” filter won’t pick it up. That is by design: Copy activity and dataset filters don’t expose a “createdDatetime” filter. See the connector docs for File System and ADLS Gen2, which list only last-modified filtering (File System, ADLS Gen2).
Quick solution
- Best if your landing zone is ADLS/Blob: use a Storage Event trigger (BlobCreated). This fires when a new blob shows up (upload, copy, move), independent of its original modified date. Configure an Event trigger on your ADLS/Blob container and kick off your copy (or validation) pipeline from there. Docs: Event triggers (BlobCreated).
- If your source is a file server and you must stay in ADF: Get Metadata + ForEach + If Condition. While Copy filters can’t use creation time, Get Metadata can read a file’s created timestamp. Pattern:
- Get Metadata on the folder to list childItems.
- ForEach over the list; inside the loop, Get Metadata on each file with fieldList = ["created"].
- If Condition: @greaterOrEquals(activity('GetMetaFile').output.created, addHours(utcNow(), -24))
- True branch: Copy that file to ADLS.
- Most robust at scale: “First-seen” watermark table. On each run, enumerate files, write a control row {path, firstSeenUtc} the first time you see each path, then copy only rows with firstSeenUtc >= utcnow() - 24h. This models “arrival time” directly.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
5 Replies
Hi valjas2208,
In ADF’s file-based connectors, the built-in file filters are based on Last Modified time, not Creation time. So when you drop an old file into your “source” folder and the OS preserves its original modified timestamp, ADF’s “modifiedDatetimeStart/End” filter won’t pick it up. That is by design: Copy activity and dataset filters don’t expose a “createdDatetime” filter. See the connector docs for File System and ADLS Gen2, which list only last-modified filtering (File System, ADLS Gen2).
Quick solution
- Best if your landing zone is ADLS/Blob: use a Storage Event trigger (BlobCreated). This fires when a new blob shows up (upload, copy, move), independent of its original modified date. Configure an Event trigger on your ADLS/Blob container and kick off your copy (or validation) pipeline from there. Docs: Event triggers (BlobCreated).
- If your source is a file server and you must stay in ADF: Get Metadata + ForEach + If Condition. While Copy filters can’t use creation time, Get Metadata can read a file’s created timestamp. Pattern:
- Get Metadata on the folder to list childItems.
- ForEach over the list; inside the loop, Get Metadata on each file with fieldList = ["created"].
- If Condition: @greaterOrEquals(activity('GetMetaFile').output.created, addHours(utcNow(), -24))
- True branch: Copy that file to ADLS.
- Most robust at scale: “First-seen” watermark table. On each run, enumerate files, write a control row {path, firstSeenUtc} the first time you see each path, then copy only rows with firstSeenUtc >= utcnow() - 24h. This models “arrival time” directly.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Thank you tayloramy for your reponse. The get metadata solution should work for my use case.
I could not understand the "first-seen" solution. Could you please elaborate more on this?
Hi valjas2208,
The first seen watermark is logging each time you see a new file name, and capturing when you saw it to fake a "created date".
This only works if your files have different names every time they are dropped.If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi valjas2208,
Thank you for using the Microsoft Fabric Community Forum. Also, thanks to tayloramy, for his inputs on this thread.
I understand the challenge when you move or copy an older file into the source folder, its last modified date does not update, so your current filter in ADF doesn’t recognize it as “new.” In this case, the creation date would be more accurate for what you want, since it reflects when the file was created in that folder.At the moment, the Copy activity in ADF only supports filtering by last modified time. To work around this, you can use the Get Metadata activity (which supports both created and lastModified fields for many storage types). A common pattern is mentioned below:
Use Get Metadata on the source folder with childItems to list files. For each file, use another Get Metadata to fetch its created timestamp. Compare that timestamp against your 24-hour window (e.g. @greaterOrEquals(item().created, addHours(utcNow(), -24))). Only copy files that meet this condition.
This way, even if an older file is dropped into the folder, its creation time in that folder will make it count as “new” and your pipeline can pick it up. For reference, here are the docs you may find helpful: https://learn.microsoft.com/en-us/fabric/data-factory/get-metadata-activity
Hope this helps you move forward. Please give it a try and let us know how it goes.
Thanks for using the Microsoft Fabric Community Forum.Strongly suggest to use Fabric CopyJobs, it can automatically do incremental file ingestion scenarios based on their created/modified dates.
