Forum Discussion

MathieuSGA's avatar
MathieuSGA
Icon for Advocate I rankAdvocate I
3 months ago
Solved

How can I read a different content from one same lakehouse table ?

I requested one Lakehouse table using 2 ways: * df1 = spark.read.parquet(f"abfss://{ws_id}@onelake.dfs.fabric.microsoft.com/{lh_id}/Tables/{table_name}") * df2 : SQL Endpoint request   I filtered...
data engineering
lakehouse
  • deborshi_nag's avatar
    deborshi_nag
    3 months ago

    Hello MathieuSGA 

     

    The underlying format of the files in /Tables would be delta, so you should use the appropriate format when reading the table. Since you're using parquet format, your Spark code is reading all parquet files that includes historical or deleted rows. 

     

    Please use: spark.read.format("delta").load(f"abfss://{ws_id}@onelake.dfs.fabric.microsoft.com/{lh_id}/Tables/{table_name}")

     

deborshi_nag's avatar
deborshi_nag
Icon for Super User rankSuper User
3 months ago

Hello MathieuSGA 

 

The underlying format of the files in /Tables would be delta, so you should use the appropriate format when reading the table. Since you're using parquet format, your Spark code is reading all parquet files that includes historical or deleted rows. 

 

Please use: spark.read.format("delta").load(f"abfss://{ws_id}@onelake.dfs.fabric.microsoft.com/{lh_id}/Tables/{table_name}")

 

MathieuSGA's avatar
MathieuSGA
Icon for Advocate I rankAdvocate I
3 months ago

Simple.
Fast.
Straight to the point.
Amazing.

Thanks for passing the knowledge.