Python Notebook read Delta Table using default lakehouse

yeah you can do this as well & convert to pandas if needed

df = spark.read.format("delta").load("/lakehouse/default/Tables/my_table")
df.show()

The reason why pyarrow was used:

• to_pyarrow_dataset()
  Loads the Delta table as a PyArrow dataset, which is a fast, columnar format. This enables efficient filtering, column pruning, and scanning of large data volumes.

• .to_table()
  Converts the Arrow dataset to an in-memory Arrow Table.

• .to_pandas()
  Finally, converts the Arrow Table to a Pandas DataFrame.

Why not just dt.to_pandas()?

Because:

• The Delta Lake Python bindings (delta-rs) are optimized for interoperability with PyArrow, not Pandas.

• They expect users to control the intermediate stages, for ex: filtering data before loading it into memory via Arrow.

Hi MangoMagic ,

Thanks for raising this in Microsoft Fabric Community.

Yes, you can read Delta tables using relative paths if your notebook is attached to the default Lakehouse. The Tables/YourTableName path works without needing to specify the full workspace or Lakehouse path.

For example by using this code

df = spark.read.format("delta").load("Tables/Sales")

Output:

This works as long as the table exists under the Tables folder of the Lakehouse mounted to the current notebook.

The full ABFS path is only required if you're accessing data from a different Lakehouse. You can copy the paths using the right-click context menu of the Tables in Lakehouse Explorer.

Load data into your lakehouse with a notebook - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

Thank you.