DeltaRuntimeException: [DELTA_STATS_COLLECTION_COLUMN_NOT_FOUND]

Hi  Surya057 ,
Thanks for using Microsoft Fabric Community.
It seems like the core issue you're facing comes from Delta trying to handle statistics for the binary column, which is causing the error. Given that you've already tried encoding the column, converting it to string, and disabling stats, the best approach now would be to separate the binary column from the rest of your data.

Recommended solutions:

1. Write the Data Without the Binary Column: First, write your main data (excluding the binary column) into the Delta table. This will allow Delta to handle everything smoothly without running into the stats collection issue with the binary data.

   df_without_binary = df.drop("My_Column_name") 
   
   df_without_binary.write.format("delta").save(delta_path)

2. Write the Binary Column Separately: Then, store the binary column by itself in a different Delta table. This keeps the binary data separate and prevents Delta from struggling with stats on that column.

3. df_binary_column = df.select("My_Column_name")
   df_binary_column.write.format("delta").save("Files/your-directory/binary-column-delta")      

4. Rejoin the Two Tables: When you need to use the binary data with the rest of the dataset, simply join the two Delta tables based on a shared key (like id).

   df_main = spark.read.format("delta").load(delta_path)
    df_binary = spark.read.format("delta").load("Files/your-directory/binary-column-delta") 
   df_final = df_main.join(df_binary, "id") 
   df_final.show()

   By writing the binary column separately, you prevent Delta from trying to process the column’s statistics (like nullcount), which is causing the error. The rest of your data can still be written to Delta without issues, and you can bring everything back together when needed by joining the two tables.
   
   

   OR
   
   

   Use Spark’s ignoreNullFields for Delta Writes: Another helpful step is to set ignoreNullFields to true. This tells Delta to ignore the nullCount for columns that have a lot of null values, which can help avoid the error.

    

   spark.conf.set("spark.databricks.delta.write.ignoreNullFields", "true")
   df.write.format("delta").save(delta_path)

   Let me know how this goes or if you need any more help implementing it!
   
   If this post helps, please consider accepting as solution and a kudos would be appreciated.
   
   Best regards,
   Vinay.