Forum Discussion
sftp file modified date
hei ShafiUllah-bc , thanks for the message.
if you understand correctly, you propose that the trigger time should be saved as a watermark - however, my idea is that i should rely on the actual modified time of the file from the sftp server, not the time i trigger a run. thus, i still need a succesfull way of accessing that particular file atribute through fabric
- ShafiUllah-bc8 hours ago
Resolver II
vladenache You're right. The watermark should be the file's own lastModified value from the SFTP server, not the time the pipeline is triggered. That last step in my previous reply was wrong on that point.
The Fabric SFTP connector supports Get Metadata, and lastModified is a valid field for a file. childItems on a folder only returns name and type, so to read the attribute you still point a second Get Metadata at each file.
1. Lookup the stored watermark, which is the max lastModified from the last successful run.
2. Get Metadata on the folder with field list Child items.
3. ForEach over @activity('Get Metadata1').output.childItems. Inside the loop, Get Metadata on @item().name with field list Last modified.
4. Copy the file only when that value is greater than the watermark, and keep the max lastModified you see.
5. After a fully successful run, write that max lastModified back as the new watermark.@greater(activity('Get File Metadata').output.lastModified, variables('LastWatermark'))If you only need to copy the new files, set Filter by last modified on the Copy source instead. Put the stored watermark in Start time (UTC) and leave End time empty. That filter uses the file's lastModified from the server, not the trigger time. Start is inclusive and end is exclusive, times are UTC in the form yyyy-mm-ddThh:mm:ss.fffZ, and the filter is ignored when File path type is List of files. Because start is inclusive, files whose lastModified equals the watermark will be selected again, which is why the condition above uses greater(). After a successful copy, still update the watermark to the max lastModified you read, not to the run time. If you use Get Metadata's own last modified filter, it enumerates every file in the folder, so avoid pointing it at a large directory. Files in subfolders are not included.
To check it, run the file-level Get Metadata on one file and compare output.lastModified with the same file's timestamp in an SFTP client. They should match, and it should not equal the activity run time.
If Get Metadata still returns the run time, the Copy filter will too, because both read the Last Modified attribute the server reports. If the notebook can reach the SFTP host, paramiko can read that attribute directly. Install paramiko if it isn't already available, and keep the password in Key Vault:import paramiko from datetime import datetime, timezone password = notebookutils.credentials.getSecret('https://<name>.vault.azure.net/', 'sftp-password') ssh = paramiko.SSHClient() ssh.load_system_host_keys() ssh.connect(hostname='<host>', username='<user>', password=password) sftp = ssh.open_sftp() print(datetime.fromtimestamp(sftp.stat('/path/to/file').st_mtime, timezone.utc)) for attr in sftp.listdir_attr('/path/to/folder'): print(attr.filename, datetime.fromtimestamp(attr.st_mtime, timezone.utc)) sftp.close() ssh.close()If paramiko or the client shows the real time and Get Metadata shows the run time, the connector is not returning the server attribute. If all of them show the run time, the server is not reporting a usable Last Modified value, and a log of processed file names is the fallback.
References:
https://learn.microsoft.com/en-us/fabric/data-factory/connector-sftp-overview
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity
https://learn.microsoft.com/en-us/fabric/data-factory/connector-sftp-copy-activity