Forum Discussion
NoteBook and delta table write/read question
Hi All.
I'm unsure if this is the right place to ask my question, so apologies if it is not.
Using PySpark
I've got some code that writes a dataframe to a delta table in an attached lakehouse.
spark_df.write.format("delta") \
.mode("overwrite") \
.saveAsTable(delta_table_name)
After... I try and read that data via
read_df = http://spark.read.format("delta").table(delta_table_name)
And I get some weird stuff back.
DataFrame[Event_Class: string, Event_Subclass: string, Current_Time: timestamp, Text_Data: string, Start_Time: timestamp, End_Time: timestamp, Duration: bigint, Cpu_Time: bigint, Success: string, Integer_Data: bigint, Object_ID: string, Table_Name: string, Partition_Name: string, Start: double, End: double]
It's not until I restart the kernel and rerun the read to I get actual rows and columns back.
What's going on? How can I ensure I get data back from my query without a kernel restart (or session stop/start)?
- David
I believe I have solved this.
I had the following in the code:
from IPython.display import display
This meant that when I called display(df) it gave me some output that was unexpected.
When I stopped and started the kernel and ran a separate code block to "do the same thing" the import was not rerun and I got different results.
Learning...
1 Reply
- dgwilsonResolver III
I believe I have solved this.
I had the following in the code:
from IPython.display import display
This meant that when I called display(df) it gave me some output that was unexpected.
When I stopped and started the kernel and ran a separate code block to "do the same thing" the import was not rerun and I got different results.
Learning...