Forum Discussion
Copy activity - Mapping column from source to destination
Hi everyone,
I have a copy activity that dynamically ingests different tables from PostgreSQL using a ForEach loop driven by a lookup table. In the Source tab I also define an additional column called ingestion_time_utc using @utcnow().
After losing all my connections, the mapping was also reset. Without any explicit mapping configured, all source columns copy successfully to the Lakehouse destination — but ingestion_time_utc is not being updated. The activity input confirms the column is created correctly with the right timestamp, but when I query the bronze table I still see the old value from before the connection loss.
My question: what is the correct way to set up mapping for this scenario? Since the pipeline is dynamic (table name and schema come from variables), Import schemas does not work at design time.
Any tips on how to configure this properly would be appreciated. Thanks!
Hi robertozsr
Thank you for contacting the Microsoft Fabric community forum.
When using a metadata-driven Copy Activity where table names and schemas change dynamically, such as in a ForEach loop, it's best to use the default (auto) mapping by leaving the Mapping tab empty. This lets Copy Activity automatically map all source columns by name, supports schema drift, and ensures columns added under Source → Additional columns (like ingestion_time_utc = @utcnow()) are written to the destination without needing explicit mapping. Adding explicit mapping switches to explicit mapping mode, which only writes mapped columns and requires full schema maintenance, disrupting dynamic behavior. If ingestion_time_utc is not updating as expected, it's usually due to the write operation: append updates the timestamp only for new rows, while overwrite or truncate-and-load refreshes the timestamp for all rows.
Here is the reference: Schema and data type mapping in copy activity - Azure Data Factory & Azure Synapse | Microsoft Learn
If you have any further questions, feel free to reach out and we'll be glad to assist.
Regards,
Microsoft Fabric Community Support Team.
2 Replies
Hi robertozsr
Thank you for contacting the Microsoft Fabric community forum.
When using a metadata-driven Copy Activity where table names and schemas change dynamically, such as in a ForEach loop, it's best to use the default (auto) mapping by leaving the Mapping tab empty. This lets Copy Activity automatically map all source columns by name, supports schema drift, and ensures columns added under Source → Additional columns (like ingestion_time_utc = @utcnow()) are written to the destination without needing explicit mapping. Adding explicit mapping switches to explicit mapping mode, which only writes mapped columns and requires full schema maintenance, disrupting dynamic behavior. If ingestion_time_utc is not updating as expected, it's usually due to the write operation: append updates the timestamp only for new rows, while overwrite or truncate-and-load refreshes the timestamp for all rows.
Here is the reference: Schema and data type mapping in copy activity - Azure Data Factory & Azure Synapse | Microsoft Learn
If you have any further questions, feel free to reach out and we'll be glad to assist.
Regards,
Microsoft Fabric Community Support Team.
Thanks, that confirm what I knew: Indeed the mapping tab was empty and it was dynamically ingest the schema.
The problem was that there was a trailing space in my input ingestion_time, and even though I fixed it few days ago. I have noticed with the json file that was keeping pass the input with trailing space even if I updated the copy activity. Therefore what is needed is to rewrite the input. Save and refresh, and being sure that the json file of the pipeline is updated on the last modified date, because (that is my assumption) it will keep read the old version for some reason. Was quite and headache for something so small
