Forum Discussion
Error Writing DataFrame to Warehouse via synapsesql – Worked Previously
Hi Anonymous,
here is the Doc. for OneLake SAS Key.
https://learn.microsoft.com/en-us/fabric/onelake/onelake-shared-access-signature-overview
https://learn.microsoft.com/en-us/fabric/onelake/how-to-create-a-onelake-shared-access-signature
I think that helps you.
Best regards
Super UserHi Anonymous
Due to change into synapsesql, means that synapse sql should not be used going forward ( depreciate ) we need to use Databricks Delta Lake Open Source table in broze+ silver warehouse. synapse sql is using data lake whereas databricks delta lake is using _delta_log( Transaction log ) with parquet files. You should not use spark notebook for everything.
Python --> Apache Spark ( Data Lake ) --> Databricks Delta Lake ( Pyspark initially and subsequent use Spark SQL )
Also, Lakehouse = Fabric Warehouse = Power BI / SSAS Tabular Semantic Model
Anonymous I encountered the same issue where the behavior was inconsistent. I found a reliable workaround:
Instead of: df.write.mode("overwrite").synapsesql(f"{silverWarehouse}.{tableSchema}.{silver_table}")
I switched to writing the data to a lakehouse first: df.write.format("delta").mode("overwrite")..save("abfss://[email protected]/lh.Lakehouse/Tables/schema/df")
Then, I inserted a Copy Data activity in the pipeline: Source would be lakehouse table, and Destination would be the target table in the warehouse which we original wanted to save as. To fully replace the table (instead of appending), add "TRUNCATE TABLE schema.df" as a pre-copy script under the destination tab.