Forum Discussion

lsabetta's avatar
lsabetta
Frequent Visitor
10 months ago
Solved

Spark Session configuration

Hi community,   Currently I'm working with an F32 based in Europe.  I have a notebook in which I am forced to use pyspark because I have to get tables from my lakehouse - if not I would totally us...
data engineering
Spark Job Definition
  • v-venuppu's avatar
    v-venuppu
    10 months ago

    Hi lsabetta ,

    The issue happens because your Lakehouse table is stored in Delta format, not plain Parquet.
    When you overwrite the table, new Parquet files are created, but old ones remain in the folder for versioning.
    If you read the folder directly as Parquet, it loads both the old and new files - that’s why you see duplicate or outdated records.

    To fix this, make sure you read the table as a Delta table instead of as raw Parquet.
    This ensures that only the latest valid version of the data is returned, without mixing older files.

    Thank you.

anilgavhane's avatar
anilgavhane
Super User
10 months ago

lsabetta 

Yes, you can read tables from a Microsoft Fabric Lakehouse without creating a Spark session, by using Lakehouse shortcuts and native connectors in Python. Here are a few options:

 

🔹 1. Use the fabric Python SDK (Preview)

Microsoft Fabric offers a Python SDK that allows you to interact with Lakehouse tables directly from a notebook using Pandas, without spinning up Spark.

 

from fabric import LakehouseClient client = LakehouseClient(workspace_id="your_workspace_id", lakehouse_id="your_lakehouse_id") df = client.read_table("your_table_name")



 

This avoids Spark entirely and loads the table as a Pandas DataFrame.

 

🔹 2. Use REST APIs or ODBC/JDBC Connectors

You can access Lakehouse tables via:

  • REST API: For metadata and file-level access.
  • ODBC/JDBC: If your Lakehouse is exposed via SQL endpoints (like Warehouse or SQL Analytics).

These methods allow you to query structured data using Python libraries like pyodbc, sqlalchemy, or pandas.read_sql().

 

🔹 3. Export Lakehouse Tables to Files

If your Lakehouse tables are stored as Delta or Parquet files:

  • You can mount or access the file path directly using pyarrow, fastparquet, or pandas.read_parquet().

 

import pandas as pd df = pd.read_parquet("https://yourlakehouseurl/path/to/table")

 

This bypasses Spark and loads the data directly into memory.