topic Re: Lakehouse Tutorial issues creating Delta Tables with notebook in Data Engineering

Lakehouse Tutorial issues creating Delta Tables with notebook

IHPROLAN — Tue, 08 Oct 2024 12:57:10 GMT

Hey Everyone,

I am following the End-to-end Lake House Tutorial under Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn .

When running the first Notebook '01 - Create Delta Tables' I get an Analysis Exception for creating the dimensions tables.

How can I solve this issue?

'AnalysisException: [DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields 'CustomerKey' and 'CustomerKey''

Thanks for your help.

Re: Lakehouse Tutorial issues creating Delta Tables with notebook

GeraldGast — Tue, 08 Oct 2024 14:27:59 GMT

Hi,

the root cause of this error are different data types in columns CustomerKey and LineageKey.

In step 3 of this turorial (https://learn.microsoft.com/en-us/fabric/data-engineering/tutorial-build-lakehouse) you import a csv to the wwilakehouse via Data Flow Gen2 and create a dimension_customer table. The second transformation step contains a transformation of field customer_key and LineageKey to bigint (int64).

Step 5 (https://learn.microsoft.com/en-us/fabric/data-engineering/tutorial-lakehouse-data-preparation) of this tutorial uses PySpark Notebooks to merge csv files into the dimension_customer table. The data types for the mentioned columns are int (int32). You can compare the data types by renaming the target file from step 3 into dimension_customer_csv and execute the data flow again.

Resulting tabel from step 3:

Resulting table from step 5:

Solution:
In step 3 of the tutorial you have to adjust the last transformation step as following:

{

{"CustomerKey", Int32.Type},

{"WWICustomerID", Int32.Type},

{"Customer", type text},

{"BillToCustomer", type text},

{"Category", type text},

{"BuyingGroup", type text},

{"PrimaryContact", type text},

{"PostalCode", type text},

{"ValidFrom", type datetime},

{"ValidTo", type datetime},

{"LineageKey", Int32.Type}

}

Safe and execute.

dimension_customer should now have int32 data types for CustomerKey, WWICustomerID and LineageKey. And this ensures, that the execution of the merge statement in PySpark notebook does not raise the mentioned error.

Re: Lakehouse Tutorial issues creating Delta Tables with notebook

dshahin — Thu, 05 Jun 2025 10:51:27 GMT

Thanks, this fixed my problem! Microsoft Fabric needs to update their documentation for this.

Re: Lakehouse Tutorial issues creating Delta Tables with notebook

Anonymous — Wed, 07 Jan 2026 10:31:35 GMT

This fixed it, thank you!