Forum Discussion

valjas2208's avatar
valjas2208
New Member
11 months ago
Solved

Get file created date

Hello everyone.   I have a ingestion pipeline to copy files from a file system to ADLS. The goal is to copy all the files that have been uploaded into the source folder within the last 24 hrs. I am...
Need Help
  • tayloramy's avatar
    tayloramy
    11 months ago

    Hi valjas2208, 

     

    In ADF’s file-based connectors, the built-in file filters are based on Last Modified time, not Creation time. So when you drop an old file into your “source” folder and the OS preserves its original modified timestamp, ADF’s “modifiedDatetimeStart/End” filter won’t pick it up. That is by design: Copy activity and dataset filters don’t expose a “createdDatetime” filter. See the connector docs for File System and ADLS Gen2, which list only last-modified filtering (File System, ADLS Gen2).

    Quick solution

    1. Best if your landing zone is ADLS/Blob: use a Storage Event trigger (BlobCreated). This fires when a new blob shows up (upload, copy, move), independent of its original modified date. Configure an Event trigger on your ADLS/Blob container and kick off your copy (or validation) pipeline from there. Docs: Event triggers (BlobCreated).
    2. If your source is a file server and you must stay in ADF: Get Metadata + ForEach + If Condition. While Copy filters can’t use creation time, Get Metadata can read a file’s created timestamp. Pattern:
      • Get Metadata on the folder to list childItems.
      • ForEach over the list; inside the loop, Get Metadata on each file with fieldList = ["created"].
      • If Condition: @greaterOrEquals(activity('GetMetaFile').output.created, addHours(utcNow(), -24))
      • True branch: Copy that file to ADLS.
      Docs: Get Metadata activity and expressions.
    3. Most robust at scale: “First-seen” watermark table. On each run, enumerate files, write a control row {path, firstSeenUtc} the first time you see each path, then copy only rows with firstSeenUtc >= utcnow() - 24h. This models “arrival time” directly.

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

v-kpoloju-msft's avatar
v-kpoloju-msft
Community Support
11 months ago

Hi valjas2208,
Thank you for using the Microsoft Fabric Community Forum. Also, thanks to tayloramy, for his inputs on this thread.

I understand the challenge when you move or copy an older file into the source folder, its last modified date does not update, so your current filter in ADF doesn’t recognize it as “new.” In this case, the creation date would be more accurate for what you want, since it reflects when the file was created in that folder. 

 

At the moment, the Copy activity in ADF only supports filtering by last modified time. To work around this, you can use the Get Metadata activity (which supports both created and lastModified fields for many storage types). A common pattern is mentioned below:

Use Get Metadata on the source folder with childItems to list files. For each file, use another Get Metadata to fetch its created timestamp. Compare that timestamp against your 24-hour window (e.g. @greaterOrEquals(item().created, addHours(utcNow(), -24))). Only copy files that meet this condition. 

This way, even if an older file is dropped into the folder, its creation time in that folder will make it count as “new” and your pipeline can pick it up. For reference, here are the docs you may find helpful: https://learn.microsoft.com/en-us/fabric/data-factory/get-metadata-activity 

Hope this helps you move forward. Please give it a try and let us know how it goes.
Thanks for using the Microsoft Fabric Community Forum.