Spark SQL data frame using in another Spark SQL dataframe

amien — Sat, 26 Oct 2024 00:17:00 GMT

I have this df

df_100 = spark.sql("""SELECT * FROM SOURCE""")

But now i want to use this df_100 as source for another df

df_200 = spark.sql("select KEYFIELD from df_100")

How can i make this work? I get an error that the table doesn't exist.

If i use this it works:

df_100.createOrReplaceTempView("df_100")

df_200 = spark.sql("select KEYFIELD from df_100")

it is possible to remove the first step (createOrReplaceTempView) or is there a better way to do this?

Re: Spark SQL data frame using in another Spark SQL dataframe

lbendlin — Sun, 27 Oct 2024 12:09:15 GMT

df_200 = df_100.select("KEYFIELD")