Forum Discussion
CopyJob - ColumnNameNotFoundInDeltaSchema - Lakehouse - additionalColumns
Hi,
My copyJob `CD_Tickets` is executed from a data pipeline on a 10 minute schedule.
Most of the time the copyJob is working fine.
But sometimes, without any changes to the data pipeline, it fails with ErrorCode=ColumnNameNotFoundInDeltaSchema
Full message:
ErrorCode=ColumnNameNotFoundInDeltaSchema,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column name: __ingestedUtc02_Staged is not present in delta table schema. Please import schema to reflect the latest.,Source=Microsoft.DataTransfer.ClientLibrary,'
The reported column `__ingestedUtc02_Staged` is an additional column to a Lakehouse datasource. (see below)
I don't think there is anything wrong with the copyJob, as it is working without any changes most of the time.
But I would like to learn what the real reason is for the execution failure.
Partial copyJob definition:
{
"name": "CD_Tickets",
"type": "Copy",
"dependsOn": [
{
"activity": "NB_AutotaskUrenSync",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"typeProperties": {
"source": {
"type": "LakehouseTableSource",
"additionalColumns": [
{
"name": "__ingestedUtc02_Staged",
"value": {
"value": "@utcNow()",
"type": "Expression"
}
}
],
"datasetSettings": {
"annotations": [],
"linkedService": {
"name": "LH_BUA_BRONZE",
"properties": {
"annotations": [],
"type": "Lakehouse",
"typeProperties": {
"workspaceId": "REDACTED",
"artifactId": "REDACTED",
"rootFolder": "Tables"
}
}
},
"type": "LakehouseTable",
"schema": [],
"typeProperties": {
"schema": "autotaskurensync",
"table": "tickets"
}
}
},
"REMOVED": "SINK, MAPPING, ETC."
}
}
Hi sannavajjala,
Thank you for your help.
Your first item in things to investigate helped me in the right direction:
- "Check whether the source Lakehouse table schema is being modified, optimized, or recreated by another process around the time the failures occur."
I investigated this with a notebook cell:
%%sql DESCRIBE HISTORY autotaskurensync.tickets;Based on the timestamp and operation columns I was able to relate ColumnNameNotFoundInDeltaSchema result to a TRUNCATE operation
operation copy data activity result CREATE OR REPLACE TABLE AS SELECT Ok TRUNCATE ColumnNameNotFoundInDeltaSchema I verified that the source table was empty when the ColumnNameNotFoundInDeltaSchema error occured.
Root cause: The copy data activity was trying to copy an emtpy table.
Assumption: The copy data activity is unable to retrieve a deltaSchema for an empty table.Solution: encapsulated the copy-data-activity with an if-activity.
4 Replies
The intermittent nature of the failure is the interesting part here. If the copy job succeeds most of the time with the same configuration, then the pipeline definition itself is probably not the root cause.
The error indicates that the Copy activity validates the source schema against the Delta table schema and, at the time of execution, it cannot find the additional column __ingestedUtc02_Staged. Since this column is being added dynamically via additionalColumns, my suspicion would be a metadata synchronization or schema caching issue rather than an actual missing column in your configuration.
A few things I'd investigate:
- Check whether the source Lakehouse table schema is being modified, optimized, or recreated by another process around the time the failures occur.
- Review whether multiple pipeline runs could be overlapping and causing temporary metadata inconsistencies.
- Open the Copy activity's Import Schema configuration and see whether refreshing the schema reduces or eliminates the occurrence.
- Review the run history for patterns (same time of day, after a notebook execution, after table maintenance operations, etc.).
Since the error references the additional column rather than one of the physical columns in the Delta table, it may also be worth testing whether moving the ingestion timestamp logic into a Dataflow, Notebook, or downstream transformation step produces more consistent behavior.
I've seen similar issues where Delta metadata and activity schema definitions temporarily become out of sync, causing sporadic failures even though subsequent retries succeed without any changes. If you can correlate the failures with another process touching the Lakehouse table, that would be my first area of investigation.
Hi sannavajjala,
Thank you for your help.
Your first item in things to investigate helped me in the right direction:
- "Check whether the source Lakehouse table schema is being modified, optimized, or recreated by another process around the time the failures occur."
I investigated this with a notebook cell:
%%sql DESCRIBE HISTORY autotaskurensync.tickets;Based on the timestamp and operation columns I was able to relate ColumnNameNotFoundInDeltaSchema result to a TRUNCATE operation
operation copy data activity result CREATE OR REPLACE TABLE AS SELECT Ok TRUNCATE ColumnNameNotFoundInDeltaSchema I verified that the source table was empty when the ColumnNameNotFoundInDeltaSchema error occured.
Root cause: The copy data activity was trying to copy an emtpy table.
Assumption: The copy data activity is unable to retrieve a deltaSchema for an empty table.Solution: encapsulated the copy-data-activity with an if-activity.
Hi marco_dwp,
Thank you for reaching out to Microsoft Fabric Community.
Thank you sannavajjala for the prompt response.
As we haven’t heard back from you, we wanted to kindly follow up to check if the response provided by the user for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa
Hi @marco_dwp,
We wanted to kindly follow up to check if the response provided by the user for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa
