Forum Discussion
spark.sql is getting old data that was deleted from Lakehouse whereas spark.read.load doesn't
I have data in a Lakehouse and I have deleted some of it. I am trying to load it from a Fabric Notebook. When I use spark.sql("SELECT * FROM parquet.`<abfs_path>/Tables/<table_name>`" then I get ...
solved by changing it to delta
spark.sql("SELECT * FROM delta.`<abfs_path>/Tables/<table_name>`")
I can't set it as the default lakehouse which is why i want to use the abfs path, how do you do this with the abfs path?
solved by changing it to delta
spark.sql("SELECT * FROM delta.`<abfs_path>/Tables/<table_name>`")