Forum Discussion

Kesahli's avatar
Kesahli
Frequent Visitor
1 year ago
Solved

Size limit on parameter value in Data Pipeline using copy data assistant?

I've used the Copy Assistant in a data pipeline to set up the ingestion of app 360 tables from one lake house to another (source lakehouse is due for decommission), but the pipeline always only loads...
data engineering
data pipeline
  • bdarbo78's avatar
    bdarbo78
    1 year ago

    Hi, I ran into the exact same issue...and was hoping to find the solution here.

    I ended up creating an array variable with a 'set variable' action and pasted the json array as its value. There does not seem to be the same limit on the variable length as with pipeline parameters. Then I used the variable in place of the initial parameter.

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi Kesahli ,

 

I have not seen specific documentation on this limitation.

 

Thank you for sharing your solution.

 

There is no other better solution, and your solution is sufficient for your needs.

 

If you want to import a large number of tables faster and better, you need to upgrade the capacity, but this is a paid action and you have realized the need so far so I don't think it's necessary.

 

Best Regards,
Yang
Community Support Team

 

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!