Microsoft Fabric Community Conference 2025, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount.
Register nowGet certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started
I have a situation where a source database is sending Parquet files. The number of files varies between 200-800. They need to be copied into a target database using Datafactory.
The problem is I don't know how many files have been sent, and I don't know if the transfer was successful. I can get the sender of the files to send me a manifest file with details of the number of files, checksum etc.
Is it possible to use Lookup or similar to verify the number of files available against the expected number of files? For instance, if the manifest says 346 files and 346 files have arrived I'll begin the copy operation in the Pipeline? But if only 345 files have arrived I will do nothing.
Is there a way to solve this?
Solved! Go to Solution.
Hi @AlanTHarris ,
There is an activity called Get Metadata that might help you:
I would do it the exact same way as you suggested:
1. Read the manifest file
2. save the "supposed" number of files from that manifest in a variable
3. check the "current" number of files in that folder with the Get Metadata activity
4. if 2. <> 3., wait for x minutes and do 3. and 4. again otherwise move onto 5.
5. start copy activity
Let me know if this works for you 🙂
/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/
Did I answer your question❓➡️ Please, mark my post as a solution ✔️ |
Also happily accepting Kudos 🙂 |
Feel free to connect with me on LinkedIn! | |
#proudtobeasuperuser | |
Hi @AlanTHarris ,
There is an activity called Get Metadata that might help you:
I would do it the exact same way as you suggested:
1. Read the manifest file
2. save the "supposed" number of files from that manifest in a variable
3. check the "current" number of files in that folder with the Get Metadata activity
4. if 2. <> 3., wait for x minutes and do 3. and 4. again otherwise move onto 5.
5. start copy activity
Let me know if this works for you 🙂
/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/
Did I answer your question❓➡️ Please, mark my post as a solution ✔️ |
Also happily accepting Kudos 🙂 |
Feel free to connect with me on LinkedIn! | |
#proudtobeasuperuser | |