FUAM Load_Items_E2E Error in Copy Data

Hi  pbi_taken,

Thank you for reaching out and providing the error details. Also thanks mabdollahi, for his inputs on this issue.

The issue arises from the creatorPrincipal column in your active_items table. This column appears to contain a complex or unsupported data type (such as a nested object or user principal), which is not supported by the Delta format in Lakehouse.

To address this, you will need to exclude the creatorPrincipal column from your data source before the pipeline processes it.

Create a new version of your table (or view) that removes the problematic column:

CREATE OR REPLACE TABLE active_items_clean AS

SELECT * EXCEPT (creatorPrincipal)

FROM active_items;

Update your Copy Data / Load_Items_E2E pipeline to point to active_items_clean instead of the original active_items table.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you for using Microsoft Community Forum.

Hi pbi_taken 
The column was added in 2025-03-01 as per https://github.com/GT-Analytics/fuam-basic/releases#:~:text=New%20column%20in,item%20more%20efficiently 

How are you loading the data? I see in copy data, can you consider Notebook instead?

If you are loading the data in notebook, then you can simply remove the column before loading to start working on the table.

df = spark.read.table("FUAM_Lakehouse.active_items")
df = df.drop("creatorPrincipal")  
df.write.format("delta").saveAsTable("TargetTable")

You can also try to convert the column in a text format before writing to another table.

I hope this answers your question.

Please feel free to mark it as correct solution if it does and kodus. 🙂 

Regards,

Mehrdad Abdollahi