Forum Discussion

dgwilson's avatar
dgwilson
Resolver III
1 year ago
Solved

NoteBook and delta table write/read question

Hi All.

 

I'm unsure if this is the right place to ask my question, so apologies if it is not.

 

Using PySpark

I've got some code that writes a dataframe to a delta table in an attached lakehouse.

 

spark_df.write.format("delta") \
    .mode("overwrite") \
    .saveAsTable(delta_table_name)

 

 

After... I try and read that data via

 

read_df = http://spark.read.format("delta").table(delta_table_name)

 

And I get some weird stuff back.

 

DataFrame[Event_Class: string, Event_Subclass: string, Current_Time: timestamp, Text_Data: string, Start_Time: timestamp, End_Time: timestamp, Duration: bigint, Cpu_Time: bigint, Success: string, Integer_Data: bigint, Object_ID: string, Table_Name: string, Partition_Name: string, Start: double, End: double]

 

 

It's not until I restart the kernel and rerun the read to I get actual rows and columns back. 

 

What's going on? How can I ensure I get data back from my query without a kernel restart (or session stop/start)?

 

- David

  • I believe I have solved this.

     

    I had the following in the code:

    from IPython.display import display


    This meant that when I called display(df) it gave me some output that was unexpected.
    When I stopped and started the kernel and ran a separate code block to "do the same thing" the import was not rerun and I got different results.

    Learning...

1 Reply

  • dgwilson's avatar
    dgwilson
    Resolver III

    I believe I have solved this.

     

    I had the following in the code:

    from IPython.display import display


    This meant that when I called display(df) it gave me some output that was unexpected.
    When I stopped and started the kernel and ran a separate code block to "do the same thing" the import was not rerun and I got different results.

    Learning...