Forum Discussion
Copy Job Not Saving
I want to know how everyone is dealing with this. I have a Copy Job that brings over 30+ tables from one lakehouse to another and has column mappings. I find that when I try to edit the mapping of one table after running the Copy Job and then hitting the Apply button, I receive a validation model error message even though I only edited a single table and then the mappings of the other tables disappear and I have to redo the mappings for all 30+ tables.
If I try to add another table from the source, all my mappings from the previous sources will disappear as well and I have to redo it. I was hoping that I could hold onto the JSON code and paste in the old mappings but there's no way to edit the Copy Job JSON file only to view it.
How is everyone editing their Copy Jobs?
7 Replies
- Thomaslleblanc
Super User
Create a support ticket with microsoft so they can log the issue and work on a,solution
- v-aatheeque
Community Support
Hi Anonymous
If applying changes is not working as expected, please ensure the recommended steps are followed when modifying the Copy Job. In some cases, re-creating or re-copying the job can help re-establish the mappings correctly.
To help us investigate further, could you please share the exact validation error message you receive when attempting to apply changes to the Copy Job? - v-aatheeque
Community Support
Hi Anonymous
We wanted to follow up to check if you’ve had an opportunity to review the previous responses. If you require further assistance, please don’t hesitate to let us know - mkjit256
Helper IV
I am having the same issue in my copy job.
i had went through the steps, and was not having any issue until the last step, save + run, where.
i am trying a watermark in read and an SCD2 in write. any sugggestions?
What i am noticing is the following:
but when i try to save and get the error above, the scd2 is gone.Update:
Seems like the watermark read and the SCD2 are not supposed to work together. - mkjit256
Helper IV
did you solve this issue or opened a ticket with MS? I am having the issue upon creation not upon edit of mapping
- Murtaza_Ghafoor
Super User
With 30+ tables and custom mappings, I would recommend stop using Copy Job as the configuration layer if there are frequent schema/mapping changes involves.
The best option in this case the ingestion to a Fabric Data Factory Pipeline + Copy activities.
The important difference is that a Copy activity exposes the mapping configuration directly and supports importing schemas and explicitly defining source → destination mappings altogether.
Add configuration at pipeline level + Copy activity + Mapping ---à Copy Activity 2 + Mapping2 -----à Copy Activity 3 + Mapping3
A pipeline Copy activity, on the other hand, has a much more conventional configuration/script representation, which is considerably more practical if you need to maintain large numbers of mappings programmatically.
If this helps, ✓ Mark as Kudos | Help Other
- SrikanthSolankiRegular Visitor
Hi Anonymous,
For 30+ tables, I would consider using a metadata-driven Fabric Data Pipeline instead of maintaining all the mappings manually in a single Copy Job.
A simple approach would be:
Metadata Table -> Lookup -> ForEach -> Copy Activity -> Destination Lakehouse
Maintain the source table, destination table, enabled flag, and load type in a metadata table. The Lookup reads the enabled tables, and the ForEach passes each table to a single Copy Activity using dynamic expressions.
If the source and destination schemas are the same, use automatic column mapping. For tables with different column names, maintain the specific mappings separately.
This makes the solution much easier to maintain. For example, if you need to add a 31st table, you only add it to the metadata table instead of modifying the existing 30+ table mappings.
You can also add a logging table to track the status, row count, duration, and errors for each table.
This approach should help avoid the issue where existing mappings need to be recreated whenever a new table is added or an existing table is modified in the Copy Job.