Pyspark notebook connect to MS Fabric Warehouse to Insert/update and get the data

[AravindPeddola]

Hi Team,

I have requirements that, I need to insert/update and select data from the MS Fabric warehouse.

I am not finding relevant documents on this, can someone help me in this?

Thanks in advance!

Aravind

[AravindPeddola]

Thanks for the update.

But when we can expect this feature( using Pyspark notebooks)

[nilendraFabric · Accepted solution]

Hi @AravindPeddola 
To stay informed about the latest updates on this feature please book these urls :

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector

https://community.fabric.microsoft.com/t5/Data-Engineering/Spark-Data-Warehouse-Connector-and-Pyspar...

https://support.fabric.microsoft.com/sv-se/blog/microsoft-fabric-june-2024-update?ft=Data-warehouse:...

I can't commit on exact dates but MS is working on this feature. 

please accept this solution and give kudos to:)

it will help the community to find the answer quickly 

thanks

[nilendraFabric]

Hi @AravindPeddola 

Currently, there is no direct way to insert, update, or select data from a Microsoft Fabric Warehouse using PySpark notebooks. However, there are some workarounds and upcoming features that can help address your requirements:
1. Read data from Warehouse:
You can use the Spark connector for Fabric Data Warehouse to read data from a warehouse into a PySpark DataFrame

from com.microsoft.spark.fabric.Constants import Constants

df = spark.read.synapsesql("<warehouse name>.<schema name>.<table or view name>")

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector

Insert/Update data:
As of now, you cannot directly write to a Fabric Warehouse from a PySpark notebook. However, there are two approaches you can consider:

Write to a Lakehouse, then access from Warehouse:
Save your PySpark DataFrame to a Lakehouse table, then access it from the Warehouse.

If you need to perform insert/update operations immediately, you can use T-SQL in a notebook to interact with the Warehouse. However, this doesn’t use PySpark directly.

Please mark this post as solution and give kudos, if this is helpful.

thanks