Forum Discussion

patto_uon's avatar
patto_uon
Frequent Visitor
9 months ago
Solved

Copy activity dynamically populated additional columns error when using Expression values

I'm trying to build out a metadata driven pipeline using copy activities. Everything works fine when I attempt to pass in a dynamically generated array for the set of additional columns, such as: {...
Need Help
Pipelines
  • Anonymous's avatar
    Anonymous
    8 months ago

    Hi patto_uon ,

     

    Thanks for the clarification. To confirm, expressions such as @{formatDateTime(utcNow())} are only evaluated when they are authored directly inside the pipeline. When the same string comes from external metadata, Fabric treats it as normal text, which is why you see the expression printed instead of the evaluated value.

    So when I mentioned a “pipeline-ready expression format,” it meant that your metadata should contain the information needed to build a valid expression, but the pipeline still needs to convert that metadata into an actual expression before passing it to the Copy activity. Currently, the service cannot execute expressions coming directly from metadata files.

    To achieve a metadata-driven approach, a small preprocessing step in the pipeline is required to translate metadata into real expressions.

     

    Thank you.

Anonymous's avatar
Anonymous
Not applicable
8 months ago

Hi patto_uon ,

 

We haven’t received an update from you in some time. Could you please let us know if the issue has been resolved?
If you still require support, please let us know, we are happy to assist you.

 

Thank you.

  • patto_uon's avatar
    patto_uon
    Frequent Visitor
    8 months ago

    Hi Anonymous ,

     

    In your response below, you mentioned two available options to achieve this goal:

    1. Pre-process the metadata before assigning it to the Copy activity and convert any "type": "Expression" entries into Fabric-compatible string expressions (for example, "@{formatDateTime(utcNow())}").
    2. Update the metadata format so engineers store expressions directly in the pipeline-ready expression format rather than using the “typed expression” structure.

    I am not quite sure I understand what you mean in option (2). What is a "pipeline-ready expression format"?

     

    I assume this is not the interpolated string format "@{...}". As I mentioned in an earlier response, when I attempt to provide a metadata parameter in the following format:

    [
      {
        "name": "dsp_last_modified",
        "value": "@{formatDateTime(utcNow())}"
      }
    ]​

     Then I see the expression string, rather than the interpolated value: