Schema Mapping on Parametrized Copy Data

[david_avgarcia]

We are setting a pipeline to bring data into a fabric lakehouse from a direct DB connection source . The pipeline calls a lookup that reads a set of table names from a controller table and then it calls a pipeline inside a for each loop passing the list of tables.

The issue if that the pipeline inside the for each loop run a copy data activity. I have left schema mapping empty but it get an error when it runs for tables that dont exist in the lakehouse. I was expected that will create the table in lakehouse based onthe source columns names. 

Is there a way to do this? Even if we have to use something different from Copy Data Activities is fine?

Here is the error for reference:

Fabric pipeline execution failed, error message received from failing pipeline - 'Operation on target Copy apsabsence failed: ErrorCode=SourceColumnIsNotDefinedInDeltaMetadata,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Source column is not defined in delta metadata. Column name: bflag2, file name: . Please import schema to reflect the latest.,Source=Microsoft.DataTransfer.ClientLibrary,''



Thanks!

[v-veshwara-msft · Accepted solution]

Hi @david_avgarcia 

Thanks for reaching out to the Microsoft Fabric Community.

In a parametrized Copy activity inside a ForEach loop, leaving schema mapping empty allows auto creation only if the destination table does not already exist in the Lakehouse.

The error:

SourceColumnIsNotDefinedInDeltaMetadata

typically means the table already exists, but its Delta metadata does not include that column. Copy activity validates against the existing schema and does not automatically evolve it.

Please check:

->Whether the destination table already exists
->If it exists, whether the bflag2 column is present
->Whether the source schema changed after the initial load

Re importing schema in the Copy activity can help ensure the latest structure is reflected, but it does not update the metadata of an existing Delta table.

If the schema has changed, you can test dropping the table before the Copy runs or using overwrite.

For dynamic multi table ingestion, you may also consider Dataflow Gen2 with automatic schema settings enabled, or a Notebook based approach, which provides more control over table creation and schema handling.

Similar threads and useful resources for reference: 

Solved: Re: Fabric Issue - Microsoft Fabric Community

Copying delta table from one lakehouse to another:... - Microsoft Fabric Community

Configure Lakehouse in a copy activity - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.
Thank you.

[v-veshwara-msft]

Hi @david_avgarcia ,

Just checking in to see if you query is resolved and if any responses were helpful.
Otherwise, feel free to reach out for further assistance.

Thank you.

[v-veshwara-msft]

Hi @david_avgarcia ,

Just wanted to check if the responses provided were helpful. If further assistance is needed, please reach out.
Thank you.

[tayloramy · Accepted solution]

Hi @david_avgarcia, 

The schema mapping cannot be empty, but it can be dynamic. The way I've handled this before is to have a json object with the schema mapping details passed in as a parameter, and then use that in the schema mapping page as dynamic content. 

Here's a great video on how to do this: Master Dynamic Data Mapping in Microsoft Fabric Data Pipelines

[david_avgarcia]

Although it will be nice a dynamic auto mappings feature, this works, thank you!!

[v-veshwara-msft · Accepted solution]

Hi @david_avgarcia 

Thanks for reaching out to the Microsoft Fabric Community.

In a parametrized Copy activity inside a ForEach loop, leaving schema mapping empty allows auto creation only if the destination table does not already exist in the Lakehouse.

The error:

SourceColumnIsNotDefinedInDeltaMetadata

typically means the table already exists, but its Delta metadata does not include that column. Copy activity validates against the existing schema and does not automatically evolve it.

Please check:

->Whether the destination table already exists
->If it exists, whether the bflag2 column is present
->Whether the source schema changed after the initial load

Re importing schema in the Copy activity can help ensure the latest structure is reflected, but it does not update the metadata of an existing Delta table.

If the schema has changed, you can test dropping the table before the Copy runs or using overwrite.

For dynamic multi table ingestion, you may also consider Dataflow Gen2 with automatic schema settings enabled, or a Notebook based approach, which provides more control over table creation and schema handling.

Similar threads and useful resources for reference: 

Solved: Re: Fabric Issue - Microsoft Fabric Community

Copying delta table from one lakehouse to another:... - Microsoft Fabric Community

Configure Lakehouse in a copy activity - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.
Thank you.