Forum Discussion

woldea's avatar
woldea
Regular Visitor
7 months ago
Solved

%%tsql magic in spark notebook

I am using fabric notebook to dynamically ingest tables from lakehouse to warehouse. I need to use pySpark as notebook default language for all the cells but I want to execute SQL code on warehouse f...
data engineering
chetanhiwale's avatar
chetanhiwale
Resolver I
7 months ago

HI woldea , 
You can use Spark Synapse SQL connector for your usecase. 
adding same code for your reference. 


# IMPORTS

import com.microsoft.spark.fabric
from com.microsoft.spark.fabric.Constants import Constants

# Code
spark.read.option(Constants.DatabaseName, "<warehouse/lakeshouse name>").synapsesql("<T-SQL Query>")

You can these docs for refernce. 
https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector?tabs=pyspark


Hope this helps. Let me know if you have any other questions.