Forum Discussion
%%tsql magic in spark notebook
HI woldea ,
You can use Spark Synapse SQL connector for your usecase.
adding same code for your reference.
# IMPORTSimport com.microsoft.spark.fabric
from com.microsoft.spark.fabric.Constants import Constants
# Code
spark.read.option(Constants.DatabaseName, "<warehouse/lakeshouse name>").synapsesql("<T-SQL Query>")
You can these docs for refernce.
https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector?tabs=pyspark
Hope this helps. Let me know if you have any other questions.
Hi woldea,
You cannot mix TSQL and PySpark notebooks I believe.
You can however mix PySpark and SparkSQL. I recommend using a SparkSQL cell to do your SQL with.
Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn
Though generally, I find it best to do all your transformations in a lakehouse, which is easier to work with, and then move your data to a warehouse after it is all nice and pretty.
- Thank you tayloramy!I was able to convert my transformation into SQL and accomplished my goal in the warehouse cell. My question here is I want to run the python transformation in the same notebook but with a tsql command as the transformation impacts the warehouse. From what I read, this can be achieved using %%tsql .Spark SQL still uses spark engine and cannot execute transformation in warehouse. So, what is the use of %%tsql and how to implement it in the notebook ? What changes are needed in the notebook setting?Thanks