Forum Discussion
Getting modification time in copy activity
I need a), the original modification time. The reason is this: Distributed and unreliable systems store the source data on SFTP on Azure. The pipeline works pretty well already, but we have no monitoring for the source systems.
I have come one step close to a solution meanwhile. Like you, I have considered childItems - by itself, it's not useful, because it doesn't contain the modification time. But I can do:
GetMetadata1 -> ForEach(childItems): GetMetadata2: modTime
The outputs of each individual GetMetadata2 contain what I need. Unfortunately, ForEach doesn't gather this output.
A work-around would be to have a notebook activity inside the ForEach to store this information bit by bit in a table, but that's rather ugly.
Community ChampionCould you use Append to array variable inside the ForEach?
That way you can store the outputs of each ForEach iteration and use the data stored in the variable in other places in the pipeline.
Do you want to copy only the files where the modified time satisfies some condition? If so, maybe you can use an IF condition in the pipeline, possibly inside the ForEach activity.
Or you want to copy all the files, and just store the modified time also, and then the notebook from Files -> Table will implement logic to use only the files which satisfies some modified time condition?