Forum Discussion
Helper IInsert/update records in warehouse with notebook
Hi,
I noticed that you can add warehouses to notebooks now. I can read data from the warehouse, but I can't seem to find the way to write to the warehouse from a notebook. Is this still not supported?
I'm logged in with the workspace admin account and I get a 403 Forbidden error.
Hi Jessy_D,
You can write data to a warehouse from a PySpark notebook, but not in the same way that you write data to a lakehouse.
I believe you need to use the warehouse sql endpoint to interact with it.
https://learn.microsoft.com/en-us/fabric/data-engineering/author-tsql-notebook
So you can't use the standard .write on a spark dataframe. But using TSQL or synapsesql you can interact with the warehouse.
3 Replies
- tayloramy
Super User
Hi Jessy_D,
You can write data to a warehouse from a PySpark notebook, but not in the same way that you write data to a lakehouse.
I believe you need to use the warehouse sql endpoint to interact with it.
https://learn.microsoft.com/en-us/fabric/data-engineering/author-tsql-notebook
So you can't use the standard .write on a spark dataframe. But using TSQL or synapsesql you can interact with the warehouse.
Jessy_DHave you tried this? Do upserts work? The notebook is linked to a lakehouse and a warehouse now. And I want to go from the silver layer to the gold layer.
- tayloramy
Hi Jessy_D,
I've had no problems with this approach so far, though I will admit that I am only doing it for one of my processes. All of my other processes were built before this existed, so I use a pipeline with a copy data activity to move data from my lakehouses to warehouses.
Give it a shot and if it doesn't work, come back here and we can troubleshoot more.