Forum Discussion
Copy Job: Using variable library prevents editing mappings
Hi PanuO, you mention that you created a Copy Job, bu then later you say "Pipeline works". Are you calling your Copy Job from a pipeline?
Assuming that this is just a misphrasing, I did a simple experiement and created a Copy Job that loads from a csv file into a Lakehouse table. When creating the Copy Job I chose "not to run immediately". Once the CJ was saved I clicked on the destination settings and parameterized the Lakehouse ID by using a Varibale from the library. My variable had a default value.
Then, I verified the mapping that I customized when creating a Job. Now, when I changed the destination Lakehouse ID to a varible the mapping completely disappeared. So I recreated it again using "import schemas".
I applied tha changes to the CJ and clicked Run. The job completed without errors.
That said, I was not able to reproduce your error.
But, I confirm that the experience with editing the Copy Job parameters especially mapping, is very confusing and not straight forward. It is possible that you skipped or missed a step in configuration and you might need that again.
On another hand, I, personally, don't see a scenraio when I would need to parameterize a Copy Job considering that it is not a fully-featured ETL and I would only use it occasionally for individual data sets. It absolutely makes sense to parameterize a data pipeline, but I would not use a Copy Job in a pipeline, instead I would use a Copy Activity that is easily and fully parameterizable. This official guide may help decide which tool to use in which situation Fabric decision guide - copy activity, dataflow, Eventstream, or Spark - Microsoft Fabric | Microsoft Learn
If you find this answer useful or solving your problem please consider giving kudos and/or marking as a solution.
- PanuO8 months agoHelper II
Thank you for great answer. Did you try to also set the workspace id? I am surprised that you are able to customize mapping as my UI just shows an error. My source is CSV file also.