Forum Discussion
Advocate IHow can I read a different content from one same lakehouse table ?
I requested one Lakehouse table using 2 ways:
* df1 = spark.read.parquet(f"abfss://{ws_id}@onelake.dfs.fabric.microsoft.com/{lh_id}/Tables/{table_name}")
* df2 : SQL Endpoint request
I filtered df1 (using `.filter(F.col('id')==1)`) and df2 (using `WHERE id = 1`).
I get this strange behavior:
* df1: >100 responses/rows
* df2: 5 responses/rows
How can this be ???
Hello MathieuSGA
The underlying format of the files in /Tables would be delta, so you should use the appropriate format when reading the table. Since you're using parquet format, your Spark code is reading all parquet files that includes historical or deleted rows.
Please use: spark.read.format("delta").load(f"abfss://{ws_id}@onelake.dfs.fabric.microsoft.com/{lh_id}/Tables/{table_name}")
2 Replies
- deborshi_nag
Super User
Hello MathieuSGA
The underlying format of the files in /Tables would be delta, so you should use the appropriate format when reading the table. Since you're using parquet format, your Spark code is reading all parquet files that includes historical or deleted rows.
Please use: spark.read.format("delta").load(f"abfss://{ws_id}@onelake.dfs.fabric.microsoft.com/{lh_id}/Tables/{table_name}")
- MathieuSGA
Simple.
Fast.
Straight to the point.
Amazing.
Thanks for passing the knowledge.
