Forum Discussion
Copy activity dynamically populated additional columns error when using Expression values
Hi patto_uon ,
Thanks for the clarification. To confirm, expressions such as @{formatDateTime(utcNow())} are only evaluated when they are authored directly inside the pipeline. When the same string comes from external metadata, Fabric treats it as normal text, which is why you see the expression printed instead of the evaluated value.
So when I mentioned a “pipeline-ready expression format,” it meant that your metadata should contain the information needed to build a valid expression, but the pipeline still needs to convert that metadata into an actual expression before passing it to the Copy activity. Currently, the service cannot execute expressions coming directly from metadata files.
To achieve a metadata-driven approach, a small preprocessing step in the pipeline is required to translate metadata into real expressions.
Thank you.
Hi patto_uon, are you using the [Set variable] activity for creating the additional columns json? If not, do the following.
- Create a pipeline variable named "AdditionalCols".
- Add a [Set variable] activity and link it to the "AdditionalCols" variable.
- Edit the dynamic content for the [Set variable] activity and add the additional columns json array as something like below.
[
{
"name": "dsp_last_modified",
"value": "@{formatDateTime(utcNow())}"
},
{
"name": "run_id",
"value": "@{pipeline().RunId}"
}
]
- In the [Copy data] activity assign the variable to the additional columns dynamics content and converting it to a json object.
@json(variables('AdditionalCols'))
Hope this helps. If so, please give a Kudos 👍 and mark as Accepted Solution ✔️.
Yes, I'm using a Set Variable activity for this particular field, and that is the workaround that I have used.
However, I'm trying to make this a generic setup for data engineers to provide additional column definitions as part of the metadata included in JSON files, so this approach won't work for that scenario.
Your initial json is from the pipeline json (with "type": "Expression"). When you want to generate a the pipeline json from metadata, you can generate the json in this format.
But if you want to input a dynamic json for additional columns in an existing pipeline, you should use the sample json code I showed you. When you edit a pipeline, have a look in the menu at View > Edit Json code. You will see the complete pipeline json code.
Hope this helps. If so, please give a Kudos 👍 and mark as Accepted Solution ✔️.
