Lakehouse Delta table changes are not reflecting in SQL Analytics Endpoint

Hi Debasish_p ,

Thanks for the reply from frithjof_v .

In addition to using Overwrite, you'll have to check that the following architectures match, and that the saves you've made have not been changed. My test was successful, here are my test steps:

Here's my raw tabular data:

When I change the column names and then save the changes using the following statement, I get an architecture mismatch error:

Use the following statement to save your changes without any problems:

df.write.format(“delta”).option(“mergeSchema”, “true”).mode(“overwrite”).save(“Tables/products”)

You can also verify that the changes were successful:

df = spark.read.format(“delta”).load(“Tables/products”)
df.show()

The last table you see at the SQL endpoint looks like this:

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

When renaming columns in Delta tables within a lakehouse, the changes reflect in SQL Endpoint analytics after a brief delay of 1-2 minutes.

Following steps worked for me, 

df = spark.sql("SELECT * FROM lakehouse_name.delta_table_name")
df = df.withColumnRenamed("old_col_name","new_col_name") 
df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("table_name")

NOTE: overwriteschema-overwrites the entire schema and data of the table, mergeschema -merges the new schema with the existing schema, ensuring that all columns (old and new) are included

After these steps, the column name change will be applied, but it may take 1-2 minutes to appear in SQL Endpoint analytics due to metadata refresh.

As far as I know, overwriteSchema is the only option, yes.

EDIT: mergeSchema is another option.