Forum Discussion

Winnie2024's avatar
Winnie2024
New Member
2 years ago
Solved

error during delta table merge with CHAR data type

I need to create one delte table. And it will be upserted many times as the time on. I have the code like below:   from delta.tables import * DeltaTable.createIfNotExists(spark) \ .tableName("t...
  • SachinNandanwar's avatar
    2 years ago

    I think it would be better to use STRING data types instead of CHAR

  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Winnie2024 

     

    Thanks to SachinNandanwar 's suggestion. Based on my test, it is very helpful! 

     

    PySpark doesn’t have a direct CHAR or VARCHAR type, you can use StringType() or "STRING" to represent the VARCHAR(15) and CHAR(3) data types.

    Try

    from delta.tables import DeltaTable
    from pyspark.sql.types import StringType
    
    # Create a Delta table with two columns
    DeltaTable.createIfNotExists(spark) \
        .tableName("test") \
        .addColumn("id", StringType()) \
        .addColumn("code", StringType()) \
        .execute()

    or

    from delta.tables import *
    from pyspark.sql.types import *
    
    DeltaTable.createIfNotExists(spark) \
        .tableName("test") \
        .addColumn("id", "STRING") \
        .addColumn("code", "STRING") \
        .execute()

    Based on my test, both of the above work for your later merge test code without any error.

     

    Best Regards,
    Jing
    If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!