Pyspark notebook : Lakehouse Sql end point

Hi msprog, 

A Fabric PySpark notebook can’t “see” T-SQL views that live in a Lakehouse’s SQL analytics endpoint via the Spark catalog. Those views are objects of the SQL endpoint (TDS/T-SQL world), not Spark. But you can query them from a notebook by connecting to the SQL endpoint (via JDBC/TDS or the built-in Fabric Spark TDS reader). Alternatively, re-create the logic as a Spark view/table if you want native Spark access. 

Query the view from a notebook

1. Get your Workspace ID and the SQL endpoint name (Lakehouse’s SQL endpoint).

2. In the notebook, use the Fabric Spark TDS reader (Scala cell) to run a T-SQL query and bring the result back as a Spark DataFrame.

// Scala cell
import com.microsoft.spark.fabric.tds.implicits.read.FabricSparkTDSImplicits._
import com.microsoft.spark.fabric.Constants

val wsId = "<your-workspace-guid>"
val lakehouseSqlEndpointName = "<your-lakehouse-sql-endpoint-name>"

// Query the view
val df = spark.read
  .option(Constants.WorkspaceId, wsId)
  .option(Constants.DatabaseName, lakehouseSqlEndpointName)
  .synapsesql("select * from dbo.YourViewName");

display(df)

Notes:

• This uses the built-in Fabric Spark TDS integration outlined in community write-ups like this walkthrough:  https://www.red-gate.com/simple-talk/blogs/fabric-query-a-sql-endpoint-from-a-notebook/

• If your query is complex and you hit parser quirks, the same article shows a prepareQuery pattern to send part of the query “as-is”.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution