topic Error loading data to LakeHouse in Data Engineering

Error loading data to LakeHouse

Ira_27 — Thu, 23 Jan 2025 03:21:10 GMT

Hello,

I am trying to do a POC on Lakehouse ETL and while using a copy activity to load data into LakeHouse i get the below error message

"{
"errorCode": "2200",
"message": "Failure happened on 'destination' side. 'Type=System.InvalidOperationException,Message=invalid partition lengths,Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError",
"target": "Copy data2",
"details": []
}

Has anyone encountered this issue before? It initially worked for 1 month of data and partition column is a date column. When i ran it again for previous month it failed with the above error.

Sink Settings

Connection: Lakehouse

Table Action: Overwrite

Enable Partitions: Yes

Parition Columns: OrderDate

Re: Error loading data to LakeHouse

Anonymous — Thu, 23 Jan 2025 12:49:37 GMT

Hi @Ira_27,

Thank you for reaching out to Microsoft Fabric Community Forum.

The error message "invalid partition lengths" you are encountering while loading data into Lakehouse suggests an issue with the partitioning configuration during the copy activity.

Please ensure that the data type of your partition column (OrderDate) is consistent and valid. If it is a date column, verify that all values adhere to the expected date format without any anomalies or incorrect formats that might cause invalid partition lengths.

Since the process initially worked for one month of data but failed for another, check for discrepancies in the data for the previous month. Look for any unexpected formats in the OrderDate column that could disrupt the partitioning logic.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!

Regards,
Vinay Pabbu

Re: Error loading data to LakeHouse

Ira_27 — Fri, 24 Jan 2025 20:48:40 GMT

@Anonymous After i posted it i figured it out about the partition incorrect format. It was using SQL datetime column as partition column and when i changed it to "yyyyMMdd" it worked fine.

Exactly like you explained! Thanks