niels_vandecoev's avatar
niels_vandecoev
New Member
2 years ago
Status:
New

BUG: spark.sql() function breaks successive comment line formatting in notebook

The code below is copied from a pyspark notebook cell. What happens is that when you use the spark.sql() function and the next line is a comment line, the comment line is not formatted green, as you...
data engineering
niels_vandecoev's avatar
niels_vandecoev
New Member
2 years ago

Here's another code snippet that the spark.sql() function seems to break, allthough the code is able to run. The code below gives the red squiggly underline errors "no viable alternative at input '\n'" and "extraneous input '.' expecting {, 'assert', 'async'...". But the code still runs. The errors can be solved by using backslashes on each line for line continuation, but that makes the code messy. When commenting or removing the spark.sql() line, the errors disappear.


df = spark.sql(query)

df1 = (df

        .withColumn("col1", lit(1))

        .select(col('*'))

    )

Recent ideas