Getting error while creating external table by using Delta Table in Apache Spark

[aAmirkHan]

Hi,

While practising in Lab 3 for DP 600 - Use Delta Table in Apache Spark. I am getting an error while creating an external table.

I am using the copy ABFS path from the File, and below is the full code.

df.write.format("delta").saveAsTable("external_products", path="abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products")

But it is giving me an error while running this code.

Py4JJavaError Traceback (most recent call last) Cell In[14], line 1 ----> 1 df.write.format("delta").saveAsTable("external_products", path="abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products") File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:1586, in DataFrameWriter.saveAsTable(self, name, format, mode, partitionBy, **options) 1584 if format is not None: 1585 self.format(format) -> 1586 self._jwrite.saveAsTable(name) File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\

Please advise what to do.

[aAmirkHan]

Thank you, yes it is working fine now.

[suparnababu8 · Accepted solution]

Hi @aAmirkHan 

I replicated your scenario.  You need to cjange your code like below example. it's working for me

df.write \
  .format("delta") \
  .option("path","abfss://Pyspark_Practise_WS@onelake.dfs.fabric.microsoft.com/rawdata_LS.Lakehouse/Files/Python_Assignment_1/Ele_Store1") \
  .saveAsTable("elec_store")

If i run this it was written to table section as delat table.

In yout case you can use this code below..with your abfss path

df.write \
  .format("delta") \
  .option("path", "abfss://4d1f6b14-075a-47b0-ab85-205195008be7@onelake.dfs.fabric.microsoft.com/8826417a-348f-488d-867a-554768d2ee1b/Files/external_products") \
  .saveAsTable("external_products")

it will works. Hope this helps you..

Thank you

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!