Forum Discussion

yongshao's avatar
yongshao
Icon for Helper III rankHelper III
2 years ago

Spark Data Warehouse Connector and Pyspark

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector At this point, Scala is the only language for this feature? any Pyspark example?   
data engineering
yongshao's avatar
yongshao
Icon for Helper III rankHelper III
2 years ago

Regarding of lakehouse SQL endpoint, since notebook is able to use spark.sql() or spark.table() to query the data of lakehouse, this new spark.read.synapsesql() is not much useful. One real use case:  use spark to automate the creation and update SQL enpoint's views without any manual work

 

frithjof_v's avatar
frithjof_v
Icon for Community Champion rankCommunity Champion
2 years ago

Yes, I think the Spark connector will mainly be useful for the Fabric Warehouse. 

 

But perhaps also some use for Lakehouse SQL Analytics Endpoint, as you mention.

 

In general, spark.sql() and %%sql cells are great ways to interact with the Lakehouse data using SQL.