Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
8 months ago
Solved

Copy Jobs Failing Because Multiple are Running

I am creating a metadata driven pipeline with a control table that cycles through a pipeline, one data object at a time. Because of the ease of use I have each data object updated through an individu...
General Question
pipelines
Tips and Tricks
  • Zanqueta's avatar
    Zanqueta
    8 months ago
    When you parameterise a Copy Data activity, you can also make the schema mapping dynamic by using metadata and expressions instead of hardcoding mappings in each pipeline.
     

    Store Mappings in a Metadata Table

    Create a control table (or JSON file) that contains:
    • Source table name
    • Target table name
    • Column mappings in JSON format, for example:
    [
  { "source": "CustomerID", "sink": "Cust_ID" },
  { "source": "CustomerName", "sink": "Cust_Name" }
]

     

    Pass Mapping to Copy Activity

    In the Copy Data activity, set the translator property dynamically:
     
     
    "translator": {
    "type": "TabularTranslator",
    "mappings": "@activity('GetMapping').output.value"
}

     

    Here, GetMapping is the Lookup activity that retrieved the JSON mapping.

     

nielsvdc's avatar
nielsvdc
Icon for Super User rankSuper User
8 months ago

Hi Anonymous, Copy Jobs are designed as standalone, scheduled tasks rather than pipeline activities, which means they don’t support orchestration features like parallel execution within a pipeline loop. That’s why you’re seeing the error that Fabric prevents multiple runs of the same Copy Job at the same time for consistency and resource management.

 

To use Copy Jobs effectively, you specify multiple tables for a single source and the Copy Job will run all the copy processes parallel to eachother. But Copy Jobs are not design with parameters, enabling them to be executed with dynamic input.

 

When you want to build a metadata driven pipeline, your options are to use a Copy Data activity or use a notebook with PySpark code. The latter one is effectively the solution with the least overhead compute.

 

Hope this helps. If so, please give kudos 👍 and mark as Accepted Solution ✔️ to help others.