Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
AlanTHarris
New Member

Only copy when all files have arrived

I have a situation where a source database is sending Parquet files. The number of files varies between 200-800. They need to be copied into a target database using Datafactory. 

 

The problem is I don't know how many files have been sent, and I don't know if the transfer was successful. I can get the sender of the files to send me a manifest file with details of the number of files, checksum etc.

 

Is it possible to use Lookup or similar to verify the number of files available against the expected number of files? For instance, if the manifest says 346 files and 346 files have arrived I'll begin the copy operation in the Pipeline? But if only 345 files have arrived I will do nothing.

 

Is there a way to solve this?

1 ACCEPTED SOLUTION
tackytechtom
Super User
Super User

Hi @AlanTHarris ,

 

There is an activity called Get Metadata that might help you:

tackytechtom_0-1713284371802.png

 

I would do it the exact same way as you suggested:

1. Read the manifest file

2. save the "supposed" number of files from that manifest in a variable

3. check the "current" number of files in that folder with the Get Metadata activity

4. if 2. <> 3., wait for x minutes and do 3. and 4. again otherwise move onto 5.
5. start copy activity

 

Let me know if this works for you 🙂

 

/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/



Did I answer your question➡️ Please, mark my post as a solution ✔️

Also happily accepting Kudos 🙂

Feel free to connect with me on LinkedIn! linkedIn

#proudtobeasuperuser 

View solution in original post

1 REPLY 1
tackytechtom
Super User
Super User

Hi @AlanTHarris ,

 

There is an activity called Get Metadata that might help you:

tackytechtom_0-1713284371802.png

 

I would do it the exact same way as you suggested:

1. Read the manifest file

2. save the "supposed" number of files from that manifest in a variable

3. check the "current" number of files in that folder with the Get Metadata activity

4. if 2. <> 3., wait for x minutes and do 3. and 4. again otherwise move onto 5.
5. start copy activity

 

Let me know if this works for you 🙂

 

/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/



Did I answer your question➡️ Please, mark my post as a solution ✔️

Also happily accepting Kudos 🙂

Feel free to connect with me on LinkedIn! linkedIn

#proudtobeasuperuser 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

Top Solution Authors