Only copy when all files have arrived

[AlanTHarris]

I have a situation where a source database is sending Parquet files. The number of files varies between 200-800. They need to be copied into a target database using Datafactory. 

The problem is I don't know how many files have been sent, and I don't know if the transfer was successful. I can get the sender of the files to send me a manifest file with details of the number of files, checksum etc.

Is it possible to use Lookup or similar to verify the number of files available against the expected number of files? For instance, if the manifest says 346 files and 346 files have arrived I'll begin the copy operation in the Pipeline? But if only 345 files have arrived I will do nothing.

Is there a way to solve this?

[tackytechtom · Accepted solution]

Hi @AlanTHarris ,

There is an activity called Get Metadata that might help you:

I would do it the exact same way as you suggested:

1. Read the manifest file

2. save the "supposed" number of files from that manifest in a variable

3. check the "current" number of files in that folder with the Get Metadata activity

4. if 2. <> 3., wait for x minutes and do 3. and 4. again otherwise move onto 5.
5. start copy activity

Let me know if this works for you 🙂

/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/