NoteBook PySpark Question from a newbie (Applying data from anthoer Dataframe in notebook)

Hi dnauflett,

You can take a load at the following code to use df.collect and df.withColumn function to achieve your requirement:

# Import modules
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit

# Create a sample DataFrame
data = [("A", 34), ("B", 45), ("C", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
display(df)

# get cellvalue from DataFrame second row, first column
cellValue = df.collect()[1][0]

# load new data
df = spark.read.format("csv").option("header","true").load("Files/churn/raw/churn.csv")

# Add a new column lit wiht extract value
df = df.withColumn("NewColumn", lit(cellValue))

display(df)

Spark dataframe: collect () vs select () - Stack Overflow

Regards,

Xiaoxin Sheng