Forum Discussion
Getting error while creating external table by using Delta Table in Apache Spark
Hi,
While practising in Lab 3 for DP 600 - Use Delta Table in Apache Spark. I am getting an error while creating an external table.
I am using the copy ABFS path from the File, and below is the full code.
df.write.format("delta").saveAsTable("external_products", path="abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products")
But it is giving me an error while running this code.
Py4JJavaError Traceback (most recent call last) Cell In[14], line 1 ----> 1 df.write.format("delta").saveAsTable("external_products", path="abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products") File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:1586, in DataFrameWriter.saveAsTable(self, name, format, mode, partitionBy, **options) 1584 if format is not None: 1585 self.format(format) -> 1586 self._jwrite.saveAsTable(name) File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\
Please advise what to do.
Hi aAmirkHan
I replicated your scenario. You need to cjange your code like below example. it's working for me
df.write \ .format("delta") \ .option("path","abfss://[email protected]/rawdata_LS.Lakehouse/Files/Python_Assignment_1/Ele_Store1") \ .saveAsTable("elec_store")If i run this it was written to table section as delat table.
In yout case you can use this code below..with your abfss path
df.write \ .format("delta") \ .option("path", "abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products") \ .saveAsTable("external_products")it will works. Hope this helps you..
Thank you
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!
3 Replies
Hi aAmirkHan
I replicated your scenario. You need to cjange your code like below example. it's working for me
df.write \ .format("delta") \ .option("path","abfss://[email protected]/rawdata_LS.Lakehouse/Files/Python_Assignment_1/Ele_Store1") \ .saveAsTable("elec_store")If i run this it was written to table section as delat table.
In yout case you can use this code below..with your abfss path
df.write \ .format("delta") \ .option("path", "abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products") \ .saveAsTable("external_products")it will works. Hope this helps you..
Thank you
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!i use your code and it can create external table.
but when i use "DESCRIBE FORMATTED external_products;", type is "MANAGED" (not "EXTERNAL"), location is "abfss://.../Tables/dbo/external_products" (not "abfss://.../Files/external_products").
Is it new feature or issues?
Thank you, yes it is working fine now.
