Forum Discussion
Get file created date
- 11 months ago
Hi valjas2208,
In ADF’s file-based connectors, the built-in file filters are based on Last Modified time, not Creation time. So when you drop an old file into your “source” folder and the OS preserves its original modified timestamp, ADF’s “modifiedDatetimeStart/End” filter won’t pick it up. That is by design: Copy activity and dataset filters don’t expose a “createdDatetime” filter. See the connector docs for File System and ADLS Gen2, which list only last-modified filtering (File System, ADLS Gen2).
Quick solution
- Best if your landing zone is ADLS/Blob: use a Storage Event trigger (BlobCreated). This fires when a new blob shows up (upload, copy, move), independent of its original modified date. Configure an Event trigger on your ADLS/Blob container and kick off your copy (or validation) pipeline from there. Docs: Event triggers (BlobCreated).
- If your source is a file server and you must stay in ADF: Get Metadata + ForEach + If Condition. While Copy filters can’t use creation time, Get Metadata can read a file’s created timestamp. Pattern:
- Get Metadata on the folder to list childItems.
- ForEach over the list; inside the loop, Get Metadata on each file with fieldList = ["created"].
- If Condition: @greaterOrEquals(activity('GetMetaFile').output.created, addHours(utcNow(), -24))
- True branch: Copy that file to ADLS.
- Most robust at scale: “First-seen” watermark table. On each run, enumerate files, write a control row {path, firstSeenUtc} the first time you see each path, then copy only rows with firstSeenUtc >= utcnow() - 24h. This models “arrival time” directly.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Strongly suggest to use Fabric CopyJobs, it can automatically do incremental file ingestion scenarios based on their created/modified dates.