Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Lakehouse - Delta table columns not updating

Hi all,   I am using a Python Notebook that's being triggered by a data factory to update Delta tables each night. In short, the process is as follows: 1. A data factory orchestrates all processes...
data engineering
AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

Hi Anonymous unfortunately using ALTER statements with Lakehouse tables causes metadata sync issues with the Lakehouse SQL Endpoint (and the semantic models) which will cause any new columns to not be visible/queryable.

 

In your scenario, are you able to use mergeSchema to evolve the schema? Delta Lake Schema Evolution | Delta Lake

Anonymous's avatar
Anonymous
Not applicable
2 years ago

That's a pity. Is this a known problem that will be fixed in the future, or is it how the Lakehouse is supposed to work?

I switched to the ALTER statements in my notebook as using mergeSchema did not work at all. I tried to use this in the first place. Source DF contained 2 new columns. I wrote it into the target delta table, and then after querying the target again, the columns were not there at all.