Fabric Delta table with column length of more than 8000 character

ThiyagarajanLG — Thu, 01 Aug 2024 09:01:12 GMT

Hi,

i tried to create the delta table in fabric as below.

df.write.mode("append").format("delta").save(fabrictablepath +"Tables/BITemporalJSONDataNew1")

The data frame (df) has the below columns.

# Create a delta table template to handle staging purpose

schema = StructType([

StructField("semID", StringType(), True),

StructField("contentType", StringType(), True),

StructField("enrichment", StringType(), True),

StructField("effectiveFrom", TimestampType(), True),

StructField("effectiveTo", TimestampType(), True),

StructField("currentFrom", TimestampType(), True),

StructField("currentTo", TimestampType(), True),

StructField("aDataItem", StringType(), True),

StructField("createdOn", StringType(), True)

])

In the dataframe, the column StructField("aDataItem", StringType(), True), can able to handle more than 8000 character length. but when i ingested the dataframe to fabric delta table, the column is truncated to 8000 character..

The question is how we can increase the length of the column in Fabric lake house delta table ??

thanks

Rajan

I tried to create some delta table in the fabric through the dataframe

Re: Fabric Delta table with column length of more than 8000 character

Anonymous — Fri, 02 Aug 2024 03:03:43 GMT

Hi ThiyagarajanLG ,

You can try to adjust the configuration of Delta Lake to ensure that long string columns are not truncated. For example, you can set the delta.dataSkippingNumIndexedCols property to avoid collecting statistics on columns that contain long values.

You can refer to below document:

Optimizations — Delta Lake Documentation

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

topic Re: Fabric Delta table with column length of more than 8000 character in Dataflow

Fabric Delta table with column length of more than 8000 character

Re: Fabric Delta table with column length of more than 8000 character