topic Re: Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUE in Data Engineering

Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUET

DebbieE — Fri, 05 Apr 2024 16:15:23 GMT

Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUET

And I assume it is going into here

I use the Code:

from delta.tables import DeltaTable

dfcont.write.format("parquet").mode("overwrite").save("Data/silver/parquet_table")

Which is where I would like the PARQUET file to sit but its just throwing errors

for example

Py4JJavaError: An error occurred while calling o6222.save. : Operation failed: "Bad Request", 400, HEAD, http://onelake.dfs.fabric.microsoft.com/cb0d4c81-7750-4e3b-9192-605359abb083/da5a0bf1-6d37-4baa-a7d6-d714fd97e60e/Data/silver/parquet_table?upn=false&action=getStatus&timeout=90

I am having an issue with this coming up too

When I go to data sources Lake houses existing Lakehouses. The Lakehouse containing the data is greyed out I cant select it. im wondering if thats part of the issue?

This is the last step and it would be great to create my first PARQUET File

Re: Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUE

smoqt — Fri, 05 Apr 2024 18:52:06 GMT

Depending on your desired result, there are different methods. The method you are using will write a parquet file to the Files location. If you want to create a Delta table, you should use the saveAsTable function with "delta" as the format.

Keep in mind that table names can only contain alphanumeric characters and underscores.

From the documentation:

# Keep it if you want to save dataframe as CSV files to Files section of the default Lakehouse df.write.mode("overwrite").format("csv").save("Files/ " + csv_table_name) # Keep it if you want to save dataframe as Parquet files to Files section of the default Lakehouse df.write.mode("overwrite").format("parquet").save("Files/" + parquet_table_name) # Keep it if you want to save dataframe as a delta lake, parquet table to Tables section of the default Lakehouse df.write.mode("overwrite").format("delta").saveAsTable(delta_table_name) # Keep it if you want to save the dataframe as a delta lake, appending the data to an existing table df.write.mode("append").format("delta").saveAsTable(delta_table_name)

Re: Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUE

Anonymous — Mon, 08 Apr 2024 08:16:07 GMT

Hi @DebbieE ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help.

Re: Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUE

DebbieE — Mon, 08 Apr 2024 14:45:35 GMT

I am confused with this

df.write.mode("overwrite").format("delta").saveAsTable(delta_table_name)

NameError: name 'DimContestant' is not defined It doesnt exist as its beand new so this doesnt work for me

Ahhhh I found out you have to do this

("DimContestant")

as an aside. Is there anyway you can include a file path or do they all just go straight into the table folder with no file path mentioned?

Re: Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUE

Anonymous — Wed, 10 Apr 2024 09:59:45 GMT

Hi @DebbieE ,

Inorder to create table, you need use the below code..

Method 1:

df.write.mode("overwrite").format("delta").saveAsTable("abc5")

In your case, you just missed to include quotes.

Method 2:

Hope this is helpful. Please let me know incase of further queries.

Re: Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUE

Anonymous — Thu, 11 Apr 2024 08:42:53 GMT

Hi @DebbieE ,

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.