topic Re: Trigger Pipeline for decrypted SFTP file - Switch broken on evaluating part of filename in Pipelines

Trigger Pipeline for decrypted SFTP file - Switch broken on evaluating part of filename

boiler87 — Sat, 15 Feb 2025 19:20:50 GMT

Hello community - New to data factory. This is probably a easy fix and would appreciate any insight.

Scenario:

File hits SFTP folder, .Net application moves the file into a decryption folder and calls a trigger based on the folder.
The first step in the triggers is "get list of files" which has a dataset that is pointing to the decrypted folder
- The file path "File Name" box in the dataset currently does not contain any dynamic content and no parameters are defined for this dataset.
- The "Get List of Files" settings are set with an argument of "Child Items" and contains a User Property of "File Name" with value "@pipeline().parameters.filename
"Get List of Files" flows to "Switch 1"
- A pipeline Parameter exists which is called 'FileName', Type: String, Default Value: @pipeline().parameters.filename
- No variables are set
- No Settings were changed
- No Output is shown
- The "Expression" box contains: "split(pipeline().parameters.FileName, '_')[1]
  - the expression box with this text causes exceptions in the "Live View Telemetry "
  - the expression "@split(pipeline().parameters.FileName, '_')[1]" causes the pipeline to fail on the swtich
- Case 1 in the switch is a "Wait 1"
- Case 2-4 are pipelines that execute on the characters in the file name:
  - Format of the file name is ######_abcdefghijk_YYYYMMDDHHSS
  - The case statements contain characters after the first "_" for example -
    - Case 1: Apple for file named ######_Apple_2025mmdd...
      - A pipeline is executed in each case and contains the following:
        Invoked Pipeline "Apple"
        Contains Parameter : FileName, Type: String, Value: "
        @activity('Get List of Files').output.itemName"
    - Case 2: Peaches for file named ######_Peaches_2025mmdd...
    - Case 3: Pumpkin for file name ######_Pumpkin_2025mmdd.
The Switch activity flows to a "Delete" activity
- The delete activity uses the same dataset in the "Get List of Files" activity above
- The file path type is "File Path in Dataset"
- Recursively box is checked
- No Logging setting
- No User Properties

Summary:

The result of a file being dropped onto the SFTP server executes the .net program, the "Get List of Files" activity, the switch executes but does not find a match on the case so passes through to the delete which cleans up the decrypted file.

I'm totally lost and know it has something to do with the way its wired. I need help and would be super appreciative of your wisdom.

Re: Trigger Pipeline for decrypted SFTP file - Switch broken on evaluating part of filename

girishthimmaiah — Sun, 16 Feb 2025 07:45:48 GMT

Your Switch activity has not the right values. One of the potential issues is that how the FileName parameter is passed or split. These are some of the things that you should test:
1. Make Sure the Filename is Actually Being Captured
Before the Switch activity, it is important to prove that the FileName parameter is assigned properly. Add a new Set Variable activity directly before the switch and write this:

@activity('Get List of Files').output.childItems[0].name
Then, execute the pipeline in Debug mode and check whether the file name appears as you expect.
2. Fix the Split Expression in Switch Activity
Right now, you are using:

split(pipeline().parameters.FileName, '_')[1]
The problem may be the file name as it may be loaded with a blank space causing the Switch to not match any cases. Adding a trim function will remove the spaces as well as other blank characters.

trim(split(pipeline().parameters.FileName, '_')[1])
Otherwise, if you’re concerned about having the wrong case, change it to lowercase explicitly:

toLower(trim(split(pipeline().parameters.FileName, '_')[1]))
This should make sure that the value matches with the case statements (Apple, Peaches, Pumpkin, etc.) exactly.
3. Double-Check the "Get List of Files" Output
During the process, go to the Debug output and check the following:
Are the filenames the correct ones that come back from it?
Do I need to loop through the files with a ForEach activity in case of multiple files?
The output may be completely empty, and the problem must have started from it!
4. Quick Fix for the Delete Activity
Despite the Switch not being able to match any folder, the file is removed anyway. It this behavior doesn’t match your expectation, a condition could be added before the Delete activity to check the right status which is true.
Next Steps
Add a Set Variable activity before the Switch → Print FileName in Debug
Fix the Split expression in the Switch
Check the Debug output of “Get List of Files”
Confirm case-sensitivity and spaces in filenames

Re: Trigger Pipeline for decrypted SFTP file - Switch broken on evaluating part of filename

boiler87 — Tue, 18 Feb 2025 11:43:47 GMT

Thank you for your detailed response!!!

I found a different approach to the problem and setup a trigger on storage of the file from the .Net job on the Decrypted folder. The trigger is on file appearing in the decrypted folder and then evulating the pipeline().paramater.filename in the switch without the need for getting metadata. This has streamlined my approach.

Again, I very much appreciate you taking the time out of your busy day to provide me with your insight.