Data Pipeline Expression Language – Check if field exists, handling non-existent properties

[sean_ms]

The expression language is a powerful tool within data pipelines that provide an extensive number of functions and features. When working with JSON objects such as a pipeline parameter of type object, or a pipeline activity’s output, you may come across a scenario where the field that you would like to reference may not exist, a non-existent property.

For example, I have a Lookup activity that is reading a JSON file.  Proceeding the Lookup, I want to capture the field, ‘missingfield’ and set the value to a variable called v_string_file_name.

The Lookup activity returns:

{ "count": 1, "value": [ { "data": [ { "id": "123" }, { "id": "456" } ] } ] }

The expression in the Set variable activity is:

@activity('Lookup1').output.value[0].missingfield

In this case, the pipeline activity will fail, returning an error message similar to:

The expression 'string(activity('Lookup1').output.value[0].missingfield)' cannot be evaluated because property 'missingfield' doesn't exist, available properties are 'data'.

To circumvent and handle this scenarios, we can use the question mark ‘?’ operator to access the non-existent property as a null value. This operator is inserted into the expression, prior to the dot notation ‘.’ that references the desired field/property. 

@activity('Lookup1').output.value[0]?.missingfield

Note: For the C# developers out there, this operator does differ from the member access and null-conditional operators and expressions, see references. 

By doing so, the Set variable activity will now succeed and return:

{"name": "v_string_file_name", "value": null}

We can now wrap the Set variable expression with the coalesce function to specify a desired value when the field does not exist.

@coalesce(activity('Lookup1').output.value[0]?.missingfield, 'Property does not exist')

The Set variable activity will now return:

{"name": "v_string_file_name", "value": "Property does not exist"}

References

Expressions and functions - Microsoft Fabric | Microsoft Learn

Reference guide for expression functions - Azure Logic Apps | Microsoft Learn

Workflow Definition Language schema reference - Azure Logic Apps | Microsoft Learn

Member access and null-conditional operators and expressions: - C# reference | Microsoft Learn

Conclusion

Thank you for taking the time to read this blog. I hope you have found this useful. Please consider giving a thumbs up and leaving a comment with questions or any recommendations for future blogs. 

Thank you @RDenkewalter for showing me this wonderful operator! 

[Rutuja1997]

Hi sean,

Indeed the pipeline expression builders equips us with an ability to use dynamic functions and parameters.However ,i came across an issue while building the folder path to the sftp server dynamically with the concat function.

Hardcoded path for example is '/Directory_name/folderName' .Now i want to dynamically pass the folderName from the the previous filter activity output which is filtering the latest folder.In the pipeline expression builder I wrote '

@concat('Directory_name/',activity('Filter1').output.Value[0].name).But getting "Expression of type: 'String' does not match the field: 'folderPath'. Expected type is 'object'." error.

I need this logic to build incremental pipeline to get the latest folder in the lakehous.

This is the filter activity payload {
"ItemsCount": 6,
"FilteredItemsCount": 1,
"Value": [
{
"name": "New_Extract_2*****64_1*************",
"type": "Folder"
}
]
}

 

Any idea on how to fix errors like these?