Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Lakehouse - Delta table columns not updating

Hi all,   I am using a Python Notebook that's being triggered by a data factory to update Delta tables each night. In short, the process is as follows: 1. A data factory orchestrates all processes...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous ,

Thanks for using Fabric Community.

As I understand the issue you're facing is that new columns and records added to a Delta Lake table through notebooks triggered by a Data Factory pipeline are not appearing when queried via the Lakehouse SQL endpoint, but they are visible in PySpark notebooks.

The Delta Lake table's internal metadata may not be fully updated, can you please execute optimization commands like OPTIMIZE or VACUUM?

Can you please try and let me know.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous 

Using the OPTIMIZE command helped with one of the two tables in which I'm experiencing this problem. The other table shows the columns in the column list in the Lakehouse explorer, but as soon as I open the preview of the Delta Table, it does not show the new columns or new records of data.

 

Maybe this has to do with the table size. The table for which OPTIMIZE did not help is quite a bit larger than the other one.