Forum Discussion

Kesahli's avatar
Kesahli
Frequent Visitor
1 year ago
Solved

Size limit on parameter value in Data Pipeline using copy data assistant?

I've used the Copy Assistant in a data pipeline to set up the ingestion of app 360 tables from one lake house to another (source lakehouse is due for decommission), but the pipeline always only loads...
data engineering
data pipeline
  • bdarbo78's avatar
    bdarbo78
    1 year ago

    Hi, I ran into the exact same issue...and was hoping to find the solution here.

    I ended up creating an array variable with a 'set variable' action and pasted the json array as its value. There does not seem to be the same limit on the variable length as with pipeline parameters. Then I used the variable in place of the initial parameter.

bdarbo78's avatar
bdarbo78
Advocate II
1 year ago

Hi, I ran into the exact same issue...and was hoping to find the solution here.

I ended up creating an array variable with a 'set variable' action and pasted the json array as its value. There does not seem to be the same limit on the variable length as with pipeline parameters. Then I used the variable in place of the initial parameter.

  • Kesahli's avatar
    Kesahli
    Frequent Visitor
    1 year ago

    That sounds like a neat solution. I'll give it a crack and see how it goes. Thanks