Notebook to read complex JSON array

Hi ToddChitt,

Wouldn't it be possible to use a couple of SQL functions like explode and col for this?

I found that suggested approach in this blog: https://medium.com/towards-data-engineering/transforming-json-to-lakehouse-tables-with-microsoft-fabric-notebooks-a-step-by-step-guide-d14458b1c284

Below is an example based on your json code in one of my test notebooks.

# Apply transformation to the dataframe
from pyspark.sql.functions import col, explode

exploded_df = df.select(explode(col("data")).alias("data"))

tf_df = exploded_df.select(

    col("data.RecordNumber").alias("RecordNumber"),
    col("data.Zipcode").alias("Zipcode")
)

display(tf_df)

dfJSON1 = tf_df.select( col("RecordNumber"), col("Zipcode"))     
dfJSON1.show()