Forum Discussion

lsabetta's avatar
lsabetta
Frequent Visitor
10 months ago
Solved

Spark Session configuration

Hi community,   Currently I'm working with an F32 based in Europe.  I have a notebook in which I am forced to use pyspark because I have to get tables from my lakehouse - if not I would totally us...
data engineering
Spark Job Definition
  • v-venuppu's avatar
    v-venuppu
    10 months ago

    Hi lsabetta ,

    The issue happens because your Lakehouse table is stored in Delta format, not plain Parquet.
    When you overwrite the table, new Parquet files are created, but old ones remain in the folder for versioning.
    If you read the folder directly as Parquet, it loads both the old and new files - that’s why you see duplicate or outdated records.

    To fix this, make sure you read the table as a Delta table instead of as raw Parquet.
    This ensures that only the latest valid version of the data is returned, without mixing older files.

    Thank you.

BhaveshPatel's avatar
BhaveshPatel
Super User
10 months ago

Hi lsabetta 

 

You should use Power BI Dataflow Gen 2. 

 

This is how it works :

Either  use Notebooks ( Python ( Pandas) + Data Lake + Delta Lake ) or Use Power BI Dataflow Gen 2 ( UI + UX).

 

Yes You can read the tables in Lakehouse using Python Pandas library. ( pd.read_tables)

  • lsabetta's avatar
    lsabetta
    Frequent Visitor
    10 months ago

    Hi BhaveshPatel ,

    Could you please give me an example of how to read the tables in a Lakehouse only with python?

    • BhaveshPatel's avatar
      BhaveshPatel
      Super User
      10 months ago

      Hi Isabella,

       

      This is how it works using Pandas:

       

      • lsabetta's avatar
        lsabetta
        Frequent Visitor
        10 months ago

        Hi BhaveshPatel 

        The problem that has reading the parquet is that if I overwrite the table, the parquet brings me the new records and the old ones.