topic Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata in Pipelines

Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

Username5328976 — Fri, 14 Jun 2024 12:48:03 GMT

Hi,

I'm trying to copy tables from one lakehouse to another. The delta tables are in lakehouse A and are there because of a dataverse<>fabric link. I just want to do a simple full copy of the table to lakehouse B, in another workspace. I use a copy data activity with lakehouse A as source (table) and lakehouse B as destination (table - overwrite). I do not configure anything else (there aren't really any options to do that anyway).

Some tables work just fine. Some tables don't and give the following error:
Source column is not defined in delta metadata

ErrorCode=SourceColumnIsNotDefinedInDeltaMetadata,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Source column is not defined in delta metadata. Column name: _change_type, file name: PartitionId=2024/part-00000-b27a9fdb-3547-4e7e-bafb-a146affa85bc.c000.snappy.parquet.,Source=Microsoft.DataTransfer.ClientLibrary,'

When I try to import schema in the "mapping" section I get the same error message.

I do not want to and cannot use shortcuts in this situation.

Why am I getting this error, and how do I solve it?

Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

v-cboorla-msft — Mon, 17 Jun 2024 05:20:23 GMT

Hi Username5328976

Thanks for using Microsoft Fabric Community.

Please refer to the following thread for similar issue which might help you : Solved: Re: How Copy files/tables from lakehouse to anothe... - Microsoft Fabric Community

I hope this information helps.

Thank you.

Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

v-cboorla-msft — Tue, 18 Jun 2024 06:37:52 GMT

Hi Username5328976

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thank you.

Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

Username5328976 — Tue, 18 Jun 2024 07:24:52 GMT

So it does work when using a pyspark notebook. That's good, but the data pipeline still gives the same error which I think should be fixed.

Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

v-cboorla-msft — Thu, 27 Jun 2024 12:23:34 GMT

Hi Username5328976

Apologies for the inconvenience and delay in response.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

v-cboorla-msft — Mon, 01 Jul 2024 05:25:02 GMT

Hi Username5328976

We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

Thank you.

Re: Copying delta table from one lakehouse to another: Source column is not defined in delta metadata

amoghshahane — Thu, 20 Feb 2025 14:54:02 GMT

Hi Username5328976 ,

Did you find a solution to this? I am facing exact same problem and funny thing is the mentioned column is not present both in source and sink. 🙂

If you did not find a solution, can you please share any workaround that you may have found useful?

Thanks,

Amogh Shahane