Forum Discussion

HenryC's avatar
HenryC
Regular Visitor
2 years ago

Error when creating table using DeltaTableBuilder API in lakehouse with schema

I was testing the lakehouse with schema feature recently. After creating the lakehouse and schema, I would like to create an empty delta table using the DeltaTableBuilder API. Here are the codes that...
data engineering
lakehouse
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi HenryC,

Have you setting and pin the default Lakehouse before you execute these codes? AFAIK, some of common libraries do not reference and initialization to the notebook environment if you not setting the default Lakehouse.

BTW, I also test with your code and it execute failed. After I doublecheck and change the created table names and it works well. (It seems not support to use 'dot' character in the table names)

from delta.tables import DeltaTable
from pyspark.sql.types import *

DeltaTable.createIfNotExists(spark) \
    .tableName("test_table1") \
    .addColumn("col1", LongType()) \
    .addColumn("col2", StringType()) \
    .addColumn("col3", StringType()) \
    .execute()

Regards,

Xiaoxin Sheng

  • HenryC's avatar
    HenryC
    Regular Visitor
    2 years ago

    Hi Anonymous , Thank you for your reply.

     

    I was testing out the lakehouse schema feature that released in July 2024. I need to specify the schema name "test" in the codes, otherwise the table will go to the default dbo schema. I probably think that it is a bug from the delta.tables libraries. 

     

     

    Henry

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi HenryC,

      OK, I reproduce this on my side.

      The notebook code processing has break (the following steps and code not executed), and the delta table seems not create correctly, so I think you can't ignore this error.

      It seems like the 'DeltaTable.createIfNotExists' function does not work with the ‘schema.tablename’ in the Lakehouse with public preview 'lakehouse schema' feature.

      You can also take a look the lakehouse schema limitations if helps:

      Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn

      Regards,

      Xiaoxin Sheng