Forum Discussion
Spark SQL vs. Spark SQL
I just learned that enclosing the code inside triple quotes """ """ also works if I want to write multi-line SQL in a PySpark cell (spark.sql-method) 😊
I'm new to this so I didn't know that from before. That is a lot simpler than adding backslack \ at the end of each line (unless someone tells me there is a keyboard shortcut to add \ to a code block ...EDIT: I just realized it's possible to hold the ALT key on the keyboard while using the mouse to click at multiple locations in the code and then insert the backslash \)
This works:
df = spark.sql("""
SELECT *
FROM orders
""")
display(df)
This works:
df = spark.sql("\
SELECT *\
FROM orders\
")
display(df)
This will not work:
df = spark.sql("
SELECT *
FROM orders
")
display(df)
Hi frithjof_v ,
I apologize for this statement - "I am curious how I can capture the output from the %%sql magic code by using the display() function or assigning the query to a variable."
As you said you can create temporary view and use it with pyspark cell is the only way.
Unfortunately, you cannot directly mix %%sql and PySpark code within a single cell in Fabric Notebooks.
Using triple quotes (""" """) is the recommended way to write multi-line SQL code within the spark.sql` function for better readability and easier line breaks.
Hope this is helpful. Please let me know incase of furher queries.
Thank you Anonymous,
I am getting a clearer understanding of what are the possibilities with both methods ☺️
Hi frithjof_v ,
I hope we have answered your query. Otherwise, will respond back with the more details and we will try to help .frithjof_v Anonymous
What are the implications for memory usage when using PySpark with a dataframe and a temp view as opposed to %%sql in a Notebook?
In other words, say your view contains billions of rows. I am assuming on the backend this is somehow on disk and not in memory.
What about when you try to load this view in a dataframe? Does the Python engine manage this intelligently or will you get an out of mem error? Or is it the notebook that handles this? Or... ?
