Anyone else getting a false error in pipelines about a column not found?

[arpost]

Greetings, all. We just started seeing a weird error with the Copy activity in a data pipeline. Here's the scenario:

We have a Copy activity that uses a Lakehouse file as its Source and a Fabric Data Warehouse as its Destination. We are adding the filename as an additional column in the Source tab:

When we run the pipeline, the data gets inserted just fine and all the pipeline steps work, BUT an error gets logged:

Here's the error message:

ErrorCode=DWCopyCommandTargetColumnNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=
Column 'FileName' not found in 'DataWarehouse' table '[stg].[file_type_a]'.,Source=Microsoft.DataTransfer.Connectors.MSSQLImport,'

See data inserted as expected with the added FileName column:

Anyone else experiencing this? I've done this kind of process with adding an additional column before without issue, so the error must have just started recently.

[arpost · Accepted solution]

Soooo.... turns out the error was legitimate. I had misaligned column names in my tables. Fabric was right again! 😂

[Anonymous]

Hi @arpost ,

Based on the error message, this appears to be a known issue, do the following: first, copy the data to the lakehouse table with type. Then, copy the data from the lakehouse table to the data warehouse.

Refer to below document:

Known issue - Pipeline fails when copying data to data warehouse with staging - Microsoft Fabric | M...

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[arpost]

Thank you for that reply, @Anonymous. This isn't actually the same error as the error to which you linked pertains to data types erroring out and failing the pipeline; the issue I reported above is a more recent thing that has started happening.

What's especially odd is I have other pipelines running that do the same operation but don't generate the "false" error; I call it false because the pipeline runs as it should and data gets inserted, but it logs an error as though it failed.