Forum Discussion

smpa01's avatar
smpa01
Community Champion
2 years ago
Solved

Underlying files have been updated. Explicitly invalidate the cache in Spark by running 'REFRESH TAB

I need to read a DW table in a NB. When I try to read this through a shortcut (of data warehouse table in a lakehouse), I am coming across following   // #1 - did not work df_one = spark.read....
lakehouse
notebook
  • Anonymous's avatar
    Anonymous
    2 years ago

    HI smpa01,

    I test on my lakehouse to create a shortcut from data warehouse, and use your codes to read data from the shortcut path. These functions works well.

    Have you setting the default the Lakehouse ? These notebook codes use some functions from library which not initialized and they will return the error. (setting default Lakehouse will attach some libraries and configurations such as the spark session)

    BTW, it seems like you are working with source data from staging table, right? AFAIK, it is a tempoar table which used  in the Extract, Transform, and Load (ETL) process within a data warehouse.

    Regards,

    Xiaoxin Sheng

Anonymous's avatar
Anonymous
Not applicable
2 years ago

HI smpa01,

I test on my lakehouse to create a shortcut from data warehouse, and use your codes to read data from the shortcut path. These functions works well.

Have you setting the default the Lakehouse ? These notebook codes use some functions from library which not initialized and they will return the error. (setting default Lakehouse will attach some libraries and configurations such as the spark session)

BTW, it seems like you are working with source data from staging table, right? AFAIK, it is a tempoar table which used  in the Extract, Transform, and Load (ETL) process within a data warehouse.

Regards,

Xiaoxin Sheng