topic Unable to create delta parquet file by using S3 shortcut in Data Engineering

Unable to create delta parquet file by using S3 shortcut

saishankar — Thu, 11 Jul 2024 20:18:11 GMT

df.write.format("delta").save("Files/bucketpocsai/parquet")

This is the code which I'm trying to run. Its giving error and i'm unable to save delta parquet file in the S3 bucket. Can anyone help me with this issue

Re: Unable to create delta parquet file by using S3 shortcut

Anonymous — Fri, 12 Jul 2024 01:55:43 GMT

Hi @saishankar ,

Ensure that you have the necessary permissions to write to S3 storage buckets and that AWS credentials are properly configured in your environment.

Try running the following code in Notebook:

df1 = spark.read.format("parquet").load("Files/AWS_1/2023/11/platform=test/") df1.write.mode("overwrite").format("delta").save("Tables/" + "New_Table")

If this method does not help you, I would like you to provide me with a detailed screenshot of the error message.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Unable to create delta parquet file by using S3 shortcut

saishankar — Fri, 12 Jul 2024 02:11:36 GMT

I have created an S3 shortcut pointing to CSV files. Then, with the Spark code, I'm attempting to store data in the same S3 bucket but in a different folder as a Delta Parquet file with the following code:

'''
df = spark.read.format("csv").option("header", True).load("Files/veritivpoc/customers100.csv")
df.write.format("delta").save("Files/veritivpoc")
```

In this setup:
- "veritivpoc" represents the S3 bucket name.

This is the error

Let me know if there's anything else you'd like to clarify or adjust!