Forum Discussion

ToddChitt's avatar
ToddChitt
Super User
2 years ago
Solved

PySpark to rename a column in a dataframe

Hello. I am quite new to Spark Notebooks. I am using one to extract JSON data to save to tables in a Lakehouse. It works, but there are some slight issues. The data, being JSON, has nexted objects. I...
data engineering
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Hi ToddChitt @can you try aliasing the column when using .select

     

    df_renamed = df.select(col("Name").alias("EmployeeName"), col("Department").alias("Dept")) 
df_renamed.show()
ToddChitt's avatar
ToddChitt
Super User
2 years ago

Hello AndyDDC and thank you for the reply. 

I tried your suggestion but it generated an error: ...name 'col; is not defined.

But from this website PySpark alias() Column & DataFrame Examples - Spark By {Examples} (sparkbyexamples.com)(which I think is about to become my new best friend 🙂 ) I added this line of code at the top of the block:

from pyspark.sql.functions import col
And that fixed it.
Another example from the site shows this syntax will work without the import statement above:
df.select ( df.Id.alias ( "Employee_Id" ),...
 
Thanks for your help.
 
  • AndyDDC's avatar
    AndyDDC
    Most Valuable Professional
    2 years ago

    Great to hear. And yes the Spark By Example website is awesome!