Copy activity delimited file dynamic settings

[g3kuser]

I am try to setup dynamic settings to a file source with file type as Delimited Text. I want to setup column and row delimiter and header row properties. I see there is an option to set dynamic expresions to setup for each setting but the expression seems to be giving an error. I tried with using variables and even that is giving similar error.

Any idea how to add dynamic expressions?

[g3kuser]

Looks to be some bug. When I added expression by double clicking on parameter name it ended up as @dataset.paramname resulting in validation errors. Instead that expression should be @pipeline().parameters.paramname. I explicitly copied the expression and it worked.

[Anonymous]

Hi @g3kuser ,

Thanks for your feedback.

Best regards,

Adamk Kong

[Sureshvarma]

Spoiler

Hi @g3kuser

Solutions and Workarounds (Until it's fixed):

1. Explicitly Type or Paste: As you've discovered, this is the most reliable workaround. Don't rely on double-clicking. Either type the full expression @pipeline().parameters.paramname manually or copy and paste it from a known-good source.

2. Use the Expression Builder (Carefully): The expression builder should work correctly. However, you need to be very careful about where you're at in the UI when you open the builder. Make sure you're in the correct property of the activity where you want to use the parameter. Sometimes, the builder might default to a dataset or linked service context, which could lead to the wrong syntax.

3. Parameterize in the Pipeline (If Possible): If you have more control over the pipeline design, try to parameterize at the pipeline level whenever possible. This often simplifies things and reduces the chances of these context issues.

4. Check JSON Definition: For complex scenarios, you can sometimes edit the underlying JSON definition of the pipeline. This requires more technical knowledge but can give you fine-grained control. Be extremely careful when editing JSON directly, as any syntax errors can break your pipeline.

5. Workaround with a Variable (Less Ideal): As a last resort, you could use a pipeline variable. Set the variable's value to @pipeline().parameters.paramname, and then use the variable in your activity. This is less direct but might work if other methods fail.

[Anonymous · Accepted solution]

Hi @g3kuser ,

In the file format setting of the Copy activity, the parameter information needs to meet the format similar to the following:

"source": {
  "type": "DelimitedTextSource",
  "storeSettings": {
    "type": "AzureBlobStorageReadSettings",
    "columnDelimiter": "@pipeline().parameters.columnDelimiter",
    "rowDelimiter": "@pipeline().parameters.rowDelimiter",
    "skipLineCount": "@pipeline().parameters.skipLineCount"
  }
}

If the problem persists, providing specific error messages and configuration snippets facilitates my further diagnosis.

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.