Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
8 months ago
Solved

Copy Jobs Failing Because Multiple are Running

I am creating a metadata driven pipeline with a control table that cycles through a pipeline, one data object at a time. Because of the ease of use I have each data object updated through an individu...
General Question
pipelines
Tips and Tricks
  • Zanqueta's avatar
    Zanqueta
    8 months ago
    When you parameterise a Copy Data activity, you can also make the schema mapping dynamic by using metadata and expressions instead of hardcoding mappings in each pipeline.
     

    Store Mappings in a Metadata Table

    Create a control table (or JSON file) that contains:
    • Source table name
    • Target table name
    • Column mappings in JSON format, for example:
    [
  { "source": "CustomerID", "sink": "Cust_ID" },
  { "source": "CustomerName", "sink": "Cust_Name" }
]

     

    Pass Mapping to Copy Activity

    In the Copy Data activity, set the translator property dynamically:
     
     
    "translator": {
    "type": "TabularTranslator",
    "mappings": "@activity('GetMapping').output.value"
}

     

    Here, GetMapping is the Lookup activity that retrieved the JSON mapping.

     

Zanqueta's avatar
Zanqueta
Icon for Super User rankSuper User
8 months ago

Hi Anonymous 

 

✅ Options to Solve This

1. Switch to Copy Activities in Pipelines

If your goal is parallelism, use Copy Activities inside a pipeline instead of Copy Jobs.
  • You can run multiple Copy Activities in parallel using ForEach with isSequential = false.
  • This gives you full control over concurrency and error handling.
Example:
 
"ForEach": {
    "items": "@activity('GetMetadata').output.value",
    "isSequential": false,
    "activities": [
        {
            "name": "CopyData",
            "type": "Copy",
            ...
        }
    ]

 

If this response was helpful in any way, I’d gladly accept a 👍much like the joy of seeing a DAX measure work first time without needing another FILTER.

Please mark it as the correct solution. It helps other community members find their way faster (and saves them from another endless loop 🌀.

 

 

  • Anonymous's avatar
    Anonymous
    Not applicable
    8 months ago

    Thank you, for parameterizing a copy data activity how are we supposed to dynamically set up the mappings? I currently have the a build a pipeline for every single copy data I want to run with the mappings within each and it's definitely not best practice

    • Zanqueta's avatar
      Zanqueta
      Icon for Super User rankSuper User
      8 months ago
      When you parameterise a Copy Data activity, you can also make the schema mapping dynamic by using metadata and expressions instead of hardcoding mappings in each pipeline.
       

      Store Mappings in a Metadata Table

      Create a control table (or JSON file) that contains:
      • Source table name
      • Target table name
      • Column mappings in JSON format, for example:
      [
  { "source": "CustomerID", "sink": "Cust_ID" },
  { "source": "CustomerName", "sink": "Cust_Name" }
]

       

      Pass Mapping to Copy Activity

      In the Copy Data activity, set the translator property dynamically:
       
       
      "translator": {
    "type": "TabularTranslator",
    "mappings": "@activity('GetMapping').output.value"
}

       

      Here, GetMapping is the Lookup activity that retrieved the JSON mapping.

       

      • v-achippa's avatar
        v-achippa
        Icon for Community Support rankCommunity Support
        8 months ago

        Hi Anonymous,

         

        Thank you for reaching out to Microsoft Fabric Community.

         

        Thank you Zanqueta and nielsvdc for the prompt response. 

         

        As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

         

        Thanks and regards,

        Anjan Kumar Chippa