Forum Discussion
Solved
NoteBook PySpark Question from a newbie (Applying data from anthoer Dataframe in notebook)
I'm probably going to have a series of questions like this!! I'm going to baby Step this. The 50,000 foot view of what I'm trying to do is, I have 524 files that I need to load into a Fabric Wareho...
Hi dnauflett,
You can take a load at the following code to use df.collect and df.withColumn function to achieve your requirement:
# Import modules from pyspark.sql import SparkSession from pyspark.sql.functions import lit # Create a sample DataFrame data = [("A", 34), ("B", 45), ("C", 29)] columns = ["Name", "Age"] df = spark.createDataFrame(data, columns) display(df) # get cellvalue from DataFrame second row, first column cellValue = df.collect()[1][0] # load new data df = spark.read.format("csv").option("header","true").load("Files/churn/raw/churn.csv") # Add a new column lit wiht extract value df = df.withColumn("NewColumn", lit(cellValue)) display(df)Spark dataframe: collect () vs select () - Stack Overflow
Regards,
Xiaoxin Sheng
Hi dnauflett,
You can take a load at the following code to use df.collect and df.withColumn function to achieve your requirement:
# Import modules
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit
# Create a sample DataFrame
data = [("A", 34), ("B", 45), ("C", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
display(df)
# get cellvalue from DataFrame second row, first column
cellValue = df.collect()[1][0]
# load new data
df = spark.read.format("csv").option("header","true").load("Files/churn/raw/churn.csv")
# Add a new column lit wiht extract value
df = df.withColumn("NewColumn", lit(cellValue))
display(df)
Spark dataframe: collect () vs select () - Stack Overflow
Regards,
Xiaoxin Sheng