Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Lakehouse - Delta table columns not updating

Hi all,   I am using a Python Notebook that's being triggered by a data factory to update Delta tables each night. In short, the process is as follows: 1. A data factory orchestrates all processes...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous ,

Thanks for using Fabric Community.

As I understand the issue you're facing is that new columns and records added to a Delta Lake table through notebooks triggered by a Data Factory pipeline are not appearing when queried via the Lakehouse SQL endpoint, but they are visible in PySpark notebooks.

The Delta Lake table's internal metadata may not be fully updated, can you please execute optimization commands like OPTIMIZE or VACUUM?

Can you please try and let me know.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hello Anonymous ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Anonymous ,

    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
    In case if you have any resolution please do share that same with the community as it can be helpful to others .
    Otherwise, will respond back with the more details and we will try to help .