Forum Discussion

Zoe_Guest's avatar
Zoe_Guest
Frequent Visitor
1 year ago
Solved

spark.sql is getting old data that was deleted from Lakehouse whereas spark.read.load doesn't

I have data in a Lakehouse and I have deleted some of it. I am trying to load it from a Fabric Notebook.   When I use spark.sql("SELECT * FROM parquet.`<abfs_path>/Tables/<table_name>`" then I get ...
lakehouse
notebook
  • Zoe_Guest's avatar
    Zoe_Guest
    1 year ago

    solved by changing it to delta

     

    spark.sql("SELECT * FROM delta.`<abfs_path>/Tables/<table_name>`")

Zoe_Guest's avatar
Zoe_Guest
Frequent Visitor
1 year ago

I can't set it as the default lakehouse which is why i want to use the abfs path, how do you do this with the abfs path?

Zoe_Guest's avatar
Zoe_Guest
Frequent Visitor
1 year ago

solved by changing it to delta

 

spark.sql("SELECT * FROM delta.`<abfs_path>/Tables/<table_name>`")