Fabric pipeline copy activity not copying multiple files

[Riskworks115]

I am having an issue with the copy activity that should not be an issue.  I am landing json files in a Fabric Lakehouse using a rest API with no issues. The pipeline checks if there are json files in the landing folder and moves them to a processed folder in a data heirarchy e.g. 2024/03/25.  There can be multiple files in the landing zone so the source is @string('path/to/jason/*.json) and the filename is empty and the file format is json.  I can see the json data when I hard code the path and use *.jason as the filename, so no permission or data problem.  There is a Set Variable prior to the If condition to set the destination path: concat('Processed/',string(formatDateTime(utcNow(),'yyyy')), '/', string(formatDateTime(utcNow(),'MM')), '/', string(formatDateTime(utcNow(),'dd'))).  Destination path is variables('DestFolder') the filename is empty and the file type is json and the Copy behavior is Flatten Heirarchy.

When I run the pipeline with the wildcard path the copy activity is successful but no data is moved.  Files read is 0 and fiels written is 0.  

In ADF this should work.  Thoughts? 

[Anonymous · Accepted solution]

Hi @Riskworks115 ,

Thanks for using Fabric Community.
I have followed below steps inorder to check - " json files in the landing folder and moves them to a processed folder in a data heirarchy e.g. 2024/03/25 ".

Steps:

1. Set Variable -

@concat('Processed/',string(formatDateTime(utcNow(),'yyyy')), '/', string(formatDateTime(utcNow(),'MM')), '/', string(formatDateTime(utcNow(),'dd')))

2. Configuration of Source in Copy Activity -

3. Configuration of Destination in Copy Activity -

After Execution of Pipeline -

Additional Ref -
3.ADF Wildcard Explained in Detail | Copy activity | Part-II (youtube.com)

Hope this is helpful. Please let me know incase of further queries.

[Anonymous · Accepted solution]

Hi @Riskworks115 ,

Thanks for using Fabric Community.
I have followed below steps inorder to check - " json files in the landing folder and moves them to a processed folder in a data heirarchy e.g. 2024/03/25 ".

Steps:

1. Set Variable -

@concat('Processed/',string(formatDateTime(utcNow(),'yyyy')), '/', string(formatDateTime(utcNow(),'MM')), '/', string(formatDateTime(utcNow(),'dd')))

2. Configuration of Source in Copy Activity -

3. Configuration of Destination in Copy Activity -

After Execution of Pipeline -

Additional Ref -
3.ADF Wildcard Explained in Detail | Copy activity | Part-II (youtube.com)

Hope this is helpful. Please let me know incase of further queries.

[Riskworks115]

Great!  Thank you, v-gchenna-msft

I must have tried everything except your solution.  Thanks again.

[Anonymous]

Hi @Riskworks115 ,

I am glad that you were able to find some insights and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

[Anonymous]

Hello @Riskworks115 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

[Anonymous · Accepted solution]

Hi @Riskworks115 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

[Anonymous]

Glad to know that your query got resolved.
Please continue using Fabric Community on your further queries.