Forum Discussion
Fabric Copy Data Pipeline Shifting Data In Columns
If your meta data changes halfway through then you cannot realistically expect this to be automated and robust. Either start over (flush and fill the parquets) or maybe implement some error handling for missing columns.
Thank you Nono & Ibendlin,
When using mapped columns, I receive the following error:
ErrorCode=ParquetColumnNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column lineid does not exist in Parquet file.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,'
I won't pretend to fully understand how the data is stored or organized in the Lakehouse, but I'm guessing there are sequention Parquet or Delta files. The first likely has fewer columns than a later file...
When loading the Lakehouse, I'm selecting * from the source system via a rest api that returns paged results in 1000 rows increments.
I could potentially define every column in my source system query, but I was realy hoping to make this entire flow dynamic and abstract the entire process to avoid creating notebooks & copy data pipeline activities for hundreds of tables.
If your meta data changes halfway through then you cannot realistically expect this to be automated and robust. Either start over (flush and fill the parquets) or maybe implement some error handling for missing columns.