Forum Discussion

DebbieE's avatar
DebbieE
Icon for Community Champion rankCommunity Champion
2 years ago
Solved

Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUET

Fabric I have created a Dataframe in Notebook using pyspark. now I want to create a Delta PARQUET

 

And I assume it is going into here

I use the Code:

 

from delta.tables import DeltaTable

dfcont.write.format("parquet").mode("overwrite").save("Data/silver/parquet_table")
 
Which is where I would like the PARQUET file to sit but its just throwing errors
 
for example
 
 
I am having an issue with this coming up too
When I go to data sources Lake houses existing Lakehouses. The Lakehouse containing the data is greyed out I cant select it. im wondering if thats part of the issue?
 
This is the last step and it would be great to create my first PARQUET File
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi DebbieE ,

    Inorder to create table, you need use the below code.. 

    Method 1:

     

    df.write.mode("overwrite").format("delta").saveAsTable("abc5")

     





    In your case, you just missed to include quotes.

    Method 2:


    Hope this is helpful. Please let me know incase of further queries.



5 Replies

  • smoqt's avatar
    smoqt
    Icon for Advocate I rankAdvocate I
    2 years ago

    Depending on your desired result, there are different methods.  The method you are using will write a parquet file to the Files location.  If you want to create a Delta table, you should use the saveAsTable function with "delta" as the format. 

     

    Keep in mind that table names can only contain alphanumeric characters and underscores.

     

    From the documentation:

     

    # Keep it if you want to save dataframe as CSV files to Files section of the default Lakehouse

df.write.mode("overwrite").format("csv").save("Files/ " + csv_table_name)

# Keep it if you want to save dataframe as Parquet files to Files section of the default Lakehouse

df.write.mode("overwrite").format("parquet").save("Files/" + parquet_table_name)

# Keep it if you want to save dataframe as a delta lake, parquet table to Tables section of the default Lakehouse

df.write.mode("overwrite").format("delta").saveAsTable(delta_table_name)

# Keep it if you want to save the dataframe as a delta lake, appending the data to an existing table

df.write.mode("append").format("delta").saveAsTable(delta_table_name)

     

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi DebbieE ,

      We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help.

    • DebbieE's avatar
      DebbieE
      Icon for Community Champion rankCommunity Champion
      2 years ago

      I am confused with this

       

      df.write.mode("overwrite").format("delta").saveAsTable(delta_table_name)

      NameError: name 'DimContestant' is not defined It doesnt exist as its beand new so this doesnt work for me

       

      Ahhhh I found out you have to do this

      ("DimContestant")
       
      as an aside. Is there anyway you can include a file path or do they all just go straight into the table folder with no file path mentioned?
       
       
       
      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi DebbieE ,

        Inorder to create table, you need use the below code.. 

        Method 1:

         

        df.write.mode("overwrite").format("delta").saveAsTable("abc5")

         





        In your case, you just missed to include quotes.

        Method 2:


        Hope this is helpful. Please let me know incase of further queries.