topic Re: Pyspark notebook : Lakehouse Sql end point in Data Engineering

Pyspark notebook : Lakehouse Sql end point

msprog — Thu, 25 Sep 2025 11:22:47 GMT

Can a pyspark notebook in Fabric connecting to the Lakehouse Sql endpoint?

Please let me know

thanks

Re: Pyspark notebook : Lakehouse Sql end point

NaveenUpadhye — Thu, 25 Sep 2025 11:38:06 GMT

No , you cant.

Lakehouse data is natively accessible from PySpark notebooks — without going through the SQL endpoint.

Re: Pyspark notebook : Lakehouse Sql end point

AntoineW — Thu, 25 Sep 2025 12:03:25 GMT

Hello msprog

Yes—use a Fabric notebook attached to the Lakehouse. You don’t need (and can’t directly “bind”) the Lakehouse SQL endpoint from PySpark; instead you query the same Delta tables via Spark.

Two simple ways:

Spark tables (recommended)

In the notebook, attach the Lakehouse (left pane → “Add lakehouse”).
Then query its tables:

# read a table registered in the Lakehouse df = spark.read.table("lakehouse.default.MyTable") # or "lakehouse.<schema>.MyTable" df.display()

# Spark SQL result = spark.sql("SELECT col1, col2 FROM lakehouse.default.MyTable WHERE col3 > 0") result.display()

Delta path (Files/Delta)

# direct Delta path under /Tables df = spark.read.format("delta").load("Tables/MyTable") df.display()

Notes

%%sql in Fabric notebooks runs Spark SQL, not T-SQL. The Lakehouse SQL analytics endpoint is for T-SQL tools (SQL editor, SSMS, Fabric items using T-SQL).
For programmatic T-SQL against a Warehouse/SQL endpoint you’d use JDBC/ODBC from outside; inside Fabric notebooks, stick to Spark/Spark SQL for Lakehouse data.

Hope it can help you !

best regards,

Antoine

Re: Pyspark notebook : Lakehouse Sql end point

msprog — Thu, 25 Sep 2025 12:42:14 GMT

thanks for this. but i want to invoke a view that is defined - i can see the view when i am on the sql endpoint. Hence i was hoping if the notebook can see the endpoint, i would be able to fire a query using the view.

Re: Pyspark notebook : Lakehouse Sql end point

tayloramy — Thu, 25 Sep 2025 13:20:30 GMT

Hi msprog,

A Fabric PySpark notebook can’t “see” T-SQL views that live in a Lakehouse’s SQL analytics endpoint via the Spark catalog. Those views are objects of the SQL endpoint (TDS/T-SQL world), not Spark. But you can query them from a notebook by connecting to the SQL endpoint (via JDBC/TDS or the built-in Fabric Spark TDS reader). Alternatively, re-create the logic as a Spark view/table if you want native Spark access.

Query the view from a notebook

Get your Workspace ID and the SQL endpoint name (Lakehouse’s SQL endpoint).
In the notebook, use the Fabric Spark TDS reader (Scala cell) to run a T-SQL query and bring the result back as a Spark DataFrame.

// Scala cell import com.microsoft.spark.fabric.tds.implicits.read.FabricSparkTDSImplicits._ import com.microsoft.spark.fabric.Constants val wsId = "<your-workspace-guid>" val lakehouseSqlEndpointName = "<your-lakehouse-sql-endpoint-name>" // Query the view val df = spark.read .option(Constants.WorkspaceId, wsId) .option(Constants.DatabaseName, lakehouseSqlEndpointName) .synapsesql("select * from dbo.YourViewName"); display(df)

Notes:

This uses the built-in Fabric Spark TDS integration outlined in community write-ups like this walkthrough: https://www.red-gate.com/simple-talk/blogs/fabric-query-a-sql-endpoint-from-a-notebook/
If your query is complex and you hit parser quirks, the same article shows a prepareQuery pattern to send part of the query “as-is”.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution

Re: Pyspark notebook : Lakehouse Sql end point

v-menakakota — Fri, 26 Sep 2025 04:16:51 GMT

Hi msprog ,
Thanks for reaching out to the Microsoft fabric community forum.

I would also take a moment to thank tayloramy , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you.

Best Regards,
Community Support Team.

Re: Pyspark notebook : Lakehouse Sql end point

Anonymous — Tue, 30 Sep 2025 06:41:35 GMT

Hello msprog,

I am also part of CST Team and we’d like to confirm whether your issue has been successfully resolved. If you still have any questions or need further assistance, please don’t hesitate to reach out. We’re more than happy to continue supporting you.

Regards,

B Manikanteswara Reddy

Re: Pyspark notebook : Lakehouse Sql end point

v-menakakota — Fri, 03 Oct 2025 06:37:01 GMT

Hi @msprog ,
Thanks for reaching out to the Microsoft fabric community forum.

I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you.

Best Regards,
Community Support Team.

Re: Pyspark notebook : Lakehouse Sql end point

BhaveshPatel — Tue, 07 Oct 2025 07:16:50 GMT

Hi msprog

Lakehouse = Fabric Data Warehouse = Power BI Semantic Model ( Direct Lake ) = Power BI Dataflow Gen 2

You can connect to lakehouse using Power BI Dataflow Gen 2 and once the table is in Delta Lake Lakehouse, You can get similar approach is Fabric Data Warehouse. Always use the Delta Lake Lakehouse Approach ( Spark ).

Yes once data is in Lakehouse, You can use the View in Fabric Data Warehouse to write SQL. or

In Lakehouse, use SHOW VIEWS and SHOW TABLES in Notebooks

Re: Pyspark notebook : Lakehouse Sql end point

smeetsh — Tue, 07 Oct 2025 03:16:34 GMT

Maybe I dont fully understand the issue so correct me if I am in the wrong direction, but in a notebook you can choose to use sparkSQL. A view is basically a stored sql script, that you call as a view in the an sql endpoint (or any sql database for that matter).

Have you tried running the actual SQL code, that makes us the view, in a sparkSQL notebook, write to a dataframe, or display it? Whatever the steps are that you need after this.

A notebook can't read the sql endpoint, but it can run SparkSQL which is quite similar to SQL.

Example:

Cheers
Hans
(if my answer is usefull, please give it a kudo or mark it as a solution)