Forum Discussion

valjas2208's avatar
valjas2208
New Member
11 months ago
Solved

Get file created date

Hello everyone.   I have a ingestion pipeline to copy files from a file system to ADLS. The goal is to copy all the files that have been uploaded into the source folder within the last 24 hrs. I am...
  • tayloramy's avatar
    11 months ago

    Hi valjas2208

     

    In ADF’s file-based connectors, the built-in file filters are based on Last Modified time, not Creation time. So when you drop an old file into your “source” folder and the OS preserves its original modified timestamp, ADF’s “modifiedDatetimeStart/End” filter won’t pick it up. That is by design: Copy activity and dataset filters don’t expose a “createdDatetime” filter. See the connector docs for File System and ADLS Gen2, which list only last-modified filtering (File System, ADLS Gen2).

    Quick solution

    1. Best if your landing zone is ADLS/Blob: use a Storage Event trigger (BlobCreated). This fires when a new blob shows up (upload, copy, move), independent of its original modified date. Configure an Event trigger on your ADLS/Blob container and kick off your copy (or validation) pipeline from there. Docs: Event triggers (BlobCreated).
    2. If your source is a file server and you must stay in ADF: Get Metadata + ForEach + If Condition. While Copy filters can’t use creation time, Get Metadata can read a file’s created timestamp. Pattern:
      • Get Metadata on the folder to list childItems.
      • ForEach over the list; inside the loop, Get Metadata on each file with fieldList = ["created"].
      • If Condition: @greaterOrEquals(activity('GetMetaFile').output.created, addHours(utcNow(), -24))
      • True branch: Copy that file to ADLS.
      Docs: Get Metadata activity and expressions.
    3. Most robust at scale: “First-seen” watermark table. On each run, enumerate files, write a control row {path, firstSeenUtc} the first time you see each path, then copy only rows with firstSeenUtc >= utcnow() - 24h. This models “arrival time” directly.

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.