Forum Discussion
Python Notebook read Delta Table using default lakehouse
yeah you can do this as well & convert to pandas if needed
df = spark.read.format("delta").load("/lakehouse/default/Tables/my_table")
df.show()The reason why pyarrow was used:
to_pyarrow_dataset()
Loads the Delta table as a PyArrow dataset, which is a fast, columnar format. This enables efficient filtering, column pruning, and scanning of large data volumes..to_table()
Converts the Arrow dataset to an in-memory Arrow Table..to_pandas()
Finally, converts the Arrow Table to a Pandas DataFrame.
Why not just dt.to_pandas()?
Because:
The Delta Lake Python bindings (delta-rs) are optimized for interoperability with PyArrow, not Pandas.
They expect users to control the intermediate stages, for ex: filtering data before loading it into memory via Arrow.
Hi MangoMagic ,
Thanks for raising this in Microsoft Fabric Community.
Yes, you can read Delta tables using relative paths if your notebook is attached to the default Lakehouse. The Tables/YourTableName path works without needing to specify the full workspace or Lakehouse path.
For example by using this code
df = spark.read.format("delta").load("Tables/Sales")Output:
This works as long as the table exists under the Tables folder of the Lakehouse mounted to the current notebook.
The full ABFS path is only required if you're accessing data from a different Lakehouse. You can copy the paths using the right-click context menu of the Tables in Lakehouse Explorer.
Load data into your lakehouse with a notebook - Microsoft Fabric | Microsoft Learn
Hope this helps. Please reach out for further assistance.
Thank you.
Super User
Yes, you can read Delta tables in Microsoft Fabric Notebooks using relative paths, similar to how you write them. The key is to reference the mounted Lakehouse path, typically under /lakehouse/default (for the default Lakehouse).
To Read a Delta Table from the Default Lakehouse (Mounted):
If the Lakehouse is already mounted in the notebook (as it usually is) you can use a relative path like
from deltalake import DeltaTable # Relative path to the delta table inside the default lakehouse relative_table_path = "/lakehouse/default/Files/tables/my_table" # Read delta table dt = DeltaTable(relative_table_path) df = dt.to_pyarrow_dataset().to_table().to_pandas() display(df)
Do not use the full Fabric workspace URL (abfss://...) unless required for advanced use cases or cross-Lakehouse access. For Tables instead of Files, use /lakehouse/default/Tables/my_table if that is where your Delta tables are written.
Thanks, for this delta table why does it need to have several intermediate functions to load into a dataframe:
dt.to_pyarrow_dataset().to_table().to_pandas()Is there a better way to read delta table into a data frame to manipulate in Python notebook?
What is "to_pyarrow_dataset"?
- Vinodh247
yeah you can do this as well & convert to pandas if needed
df = spark.read.format("delta").load("/lakehouse/default/Tables/my_table")
df.show()The reason why pyarrow was used:
to_pyarrow_dataset()
Loads the Delta table as a PyArrow dataset, which is a fast, columnar format. This enables efficient filtering, column pruning, and scanning of large data volumes..to_table()
Converts the Arrow dataset to an in-memory Arrow Table..to_pandas()
Finally, converts the Arrow Table to a Pandas DataFrame.
Why not just dt.to_pandas()?
Because:
The Delta Lake Python bindings (delta-rs) are optimized for interoperability with PyArrow, not Pandas.
They expect users to control the intermediate stages, for ex: filtering data before loading it into memory via Arrow.
