Forum Discussion
Advocate IFabric Warehouse write failure – COPY INTO staging error (13807) when using synapsesql() from Spark
This issue reproduces even with the simplest possible test case (1‑column integer table + 1‑row DataFrame).
This suggests that th
e Fabric staging directory created by the connector cannot be enumerated by the internal COPY INTO command, even though Spark successfully attempts to write the intermediate parquet files.
I have verified:
- The Warehouse exists and is accessible.
- The table exists and has matching schema.
- The Notebook and Warehouse are in the same workspace.
- Runtime: Fabric Runtime 1.3.
- The error reproduces with:
- append mode
- overwrite mode
- 1‑column table
- 1‑row DataFrame
- any data type (int / string)
- Restarting the Spark session does not consistently fix it.
from pyspark.sql import Row
df = spark.createDataFrame([Row(value_int=123)], "value_int INT")
df.write.mode("append").synapsesql("PipelineOrchestrator.dbo.demo_in
CREATE TABLE [PipelineOrchestrator].[dbo].[demo_int] (
value_int INT NULL
);
Impact:
This issue blocks all ingestion from Spark Notebook into Warehouse tables in our workspace. Even the simplest inserts fail.
Request:
Please investigate the internal COPY INTO staging process for the Fabric Spark → Warehouse connector.
I would appreciate confirmation of:
- whether this is a known backend issue,
- if there is an active incident related to Warehouse write staging,
- and whether there is a recommended workaround until a fix is deployed.
Thank you.
The problem has been resolved. Our IT department on Entra level swiched on the Private link feature...After revert back everything is working. Thank you your help guys.
7 Replies
DSZThe problem has been resolved. Our IT department on Entra level swiched on the Private link feature...After revert back everything is working. Thank you your help guys.
Hi DSZ ,
We really appreciate your efforts and for letting us know the update on the issue.
Please continue using fabric community forum for your further assistance.
Regards
- Tamanchu
Super User
Hi DSZ,
This looks like a known intermittent issue with the synapsesql connector's internal staging mechanism on Runtime 1.3. A few things to try:
- Try Runtime 1.2, Some users reported this issue is specific to Runtime 1.3. Switching to 1.2 may help as a temporary workaround.
- Use a Lakehouse as a staging area, Instead of synapsesql, write your DataFrame to a Lakehouse Delta table first, then use a CREATE TABLE AS SELECT or INSERT INTO ... SELECT from the SQL endpoint.
- Use COPY INTO manually, Write your data as Parquet to a known OneLake path, then run a COPY INTO statement via %%sql in your notebook.
If none of these work, I'd suggest opening a support ticket, this could be a backend staging issue on Microsoft's side.
Did this help?
Drop a kudo so others can find it !😉
- DSZ
Hi Tamanchu,
I think this is also could be broken probably if you have Private Link set on tennant level. I try to test this out first of all. Thank you your workarround.
- deborshi_nag
Hello DSZ
If you're reading data from a Lakehouse and upsert/merge data into a Warehouse all within the same workspace, it is best to use a TSQL Notebook, rather than PySpark Notebook using .synapsesql.
It is much easy to use a TSQL like the one below.
-- this creates the table schema on the warehouse SELECT * INTO warehouse.dbo.target_table FROM lakehouse.dbo.source_table WHERE 1 = 0; -- this upserts data into the warehouse MERGE warehouse.dbo.target_table AS T USING lakehouse.dbo.source_table AS S ON S.key_column = T.key_column WHEN NOT MATCHED THEN INSERT ( ) VALUES ( ) ;Is the lakehouse.dbo.source_table auto refreshes the data when using that table in warehouse or does it take some time to reflect latest data?
Hi DSZ
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.
