Forum Discussion
Post ProdigyAnyone else getting a false error in pipelines about a column not found?
Greetings, all. We just started seeing a weird error with the Copy activity in a data pipeline. Here's the scenario:
We have a Copy activity that uses a Lakehouse file as its Source and a Fabric Data Warehouse as its Destination. We are adding the filename as an additional column in the Source tab:
When we run the pipeline, the data gets inserted just fine and all the pipeline steps work, BUT an error gets logged:
Here's the error message:
ErrorCode=DWCopyCommandTargetColumnNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=
Column 'FileName' not found in 'DataWarehouse' table '[stg].[file_type_a]'.,Source=Microsoft.DataTransfer.Connectors.MSSQLImport,'
See data inserted as expected with the added FileName column:
Anyone else experiencing this? I've done this kind of process with adding an additional column before without issue, so the error must have just started recently.
Soooo.... turns out the error was legitimate. I had misaligned column names in my tables. Fabric was right again! 😂
3 Replies
Hi arpost ,
Based on the error message, this appears to be a known issue, do the following: first, copy the data to the lakehouse table with type. Then, copy the data from the lakehouse table to the data warehouse.
Refer to below document:
Best Regards,
Adamk KongIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
arpostThank you for that reply, Anonymous. This isn't actually the same error as the error to which you linked pertains to data types erroring out and failing the pipeline; the issue I reported above is a more recent thing that has started happening.
What's especially odd is I have other pipelines running that do the same operation but don't generate the "false" error; I call it false because the pipeline runs as it should and data gets inserted, but it logs an error as though it failed.
- arpost
Soooo.... turns out the error was legitimate. I had misaligned column names in my tables. Fabric was right again! 😂
