Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
8 months ago
Solved

Copy Jobs Failing Because Multiple are Running

I am creating a metadata driven pipeline with a control table that cycles through a pipeline, one data object at a time. Because of the ease of use I have each data object updated through an individu...
General Question
pipelines
Tips and Tricks
  • Zanqueta's avatar
    Zanqueta
    8 months ago
    When you parameterise a Copy Data activity, you can also make the schema mapping dynamic by using metadata and expressions instead of hardcoding mappings in each pipeline.
     

    Store Mappings in a Metadata Table

    Create a control table (or JSON file) that contains:
    • Source table name
    • Target table name
    • Column mappings in JSON format, for example:
    [
  { "source": "CustomerID", "sink": "Cust_ID" },
  { "source": "CustomerName", "sink": "Cust_Name" }
]

     

    Pass Mapping to Copy Activity

    In the Copy Data activity, set the translator property dynamically:
     
     
    "translator": {
    "type": "TabularTranslator",
    "mappings": "@activity('GetMapping').output.value"
}

     

    Here, GetMapping is the Lookup activity that retrieved the JSON mapping.

     

Anonymous's avatar
Anonymous
Not applicable
8 months ago

Thank you, for parameterizing a copy data activity how are we supposed to dynamically set up the mappings? I currently have the a build a pipeline for every single copy data I want to run with the mappings within each and it's definitely not best practice

Zanqueta's avatar
Zanqueta
Icon for Super User rankSuper User
8 months ago
When you parameterise a Copy Data activity, you can also make the schema mapping dynamic by using metadata and expressions instead of hardcoding mappings in each pipeline.
 

Store Mappings in a Metadata Table

Create a control table (or JSON file) that contains:
  • Source table name
  • Target table name
  • Column mappings in JSON format, for example:
[
  { "source": "CustomerID", "sink": "Cust_ID" },
  { "source": "CustomerName", "sink": "Cust_Name" }
]

 

Pass Mapping to Copy Activity

In the Copy Data activity, set the translator property dynamically:
 
 
"translator": {
    "type": "TabularTranslator",
    "mappings": "@activity('GetMapping').output.value"
}

 

Here, GetMapping is the Lookup activity that retrieved the JSON mapping.

 

  • v-achippa's avatar
    v-achippa
    Icon for Community Support rankCommunity Support
    8 months ago

    Hi Anonymous,

     

    Thank you for reaching out to Microsoft Fabric Community.

     

    Thank you Zanqueta and nielsvdc for the prompt response. 

     

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

     

    Thanks and regards,

    Anjan Kumar Chippa

    • v-achippa's avatar
      v-achippa
      Icon for Community Support rankCommunity Support
      8 months ago

      Hi @BriefStop,

       

      We wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

       

      Thanks and regards,

      Anjan Kumar Chippa