Forum Discussion
Post ProdigyUsing Spark SQL to alter a column of an existing lakehouse table
Hi pmscorca ,
Thanks for the reply from frithjof_v .
The types supported by Cast() method include: "string", "boolean", "byte", "short", "int", "long", "float", "double", "decimal", "date", "timestamp".
Here is an official document about Cast() method:
Column.Cast(String) Method (Microsoft.Spark.Sql) - .NET for Apache Spark | Microsoft Learn
In Spark SQL, there is no direct way to cast a string to a varchar with a specified length.
Here is an alternative:
You can use withColumn and cast:
from pyspark.sql.functions import col df = spark.sql("SELECT * FROM MYTABLE") df = df.withColumn("mycolumn", col("mycolumn").cast("new_data_type"))If you have any other questions, please feel free to contact me.
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Hi pmscorca ,
Thanks for the reply from frithjof_v .
The types supported by Cast() method include: "string", "boolean", "byte", "short", "int", "long", "float", "double", "decimal", "date", "timestamp".
Here is an official document about Cast() method:
Column.Cast(String) Method (Microsoft.Spark.Sql) - .NET for Apache Spark | Microsoft Learn
In Spark SQL, there is no direct way to cast a string to a varchar with a specified length.
Here is an alternative:
You can use withColumn and cast:
from pyspark.sql.functions import col
df = spark.sql("SELECT * FROM MYTABLE")
df = df.withColumn("mycolumn", col("mycolumn").cast("new_data_type"))
If you have any other questions, please feel free to contact me.
Best Regards,
Yang
Community Support Team
If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Post ProdigyHi,
I can use this PySpark code successfully:
sql_sample_alter_table = """
ALTER TABLE sql_sample_table ALTER COLUMN descr TYPE VARCHAR(100)
"""
spark.sql(sql_sample_alter_table)