Forum Discussion

vladenache's avatar
vladenache
Regular Visitor
1 day ago

sftp file modified date

Hei  Having a hard time picking up the modified datetime of files from a sftp server. With a Get MEtadata activity, the Last Modified field actually is just the time when I run that activity - so t...
data engineering
data pipeline
Prince0011's avatar
Prince0011
Icon for Memorable Member rankMemorable Member
4 hours ago

Hi,

Two things to check. If you run Get Metadata on the folder, you only get the folder’s timestamp, and some SFTP servers report that as the current time. Instead, get the childItems, loop over them with a ForEach, and run Get Metadata on each file to read its own Last Modified.

The easier route is to skip it and use the Copy activity’s “Filter by last modified” (start and end time) on the SFTP source. Store the last run time in a watermark table or pipeline variable and pass it in, so only new files get picked up.

If this helped, feel free to give it a kudos or mark it as solution, helps others find it too.