Forum Discussion

silly_bird's avatar
silly_bird
Regular Visitor
1 year ago
Solved

Help extracting value from dict in a column

  Hi all   I'm working on API integration in PySpark notebook and there is a column with email & phone that is an array with random order       contactMethods = [{'name': 'Email', 'value': 'e...
  • silly_bird's avatar
    1 year ago

    It looks like I managed to figure one solution.

    Don't know good or bad, it is my only one 

     

    from pyspark.sql.functions import col, udf
    from pyspark.sql.types import StringType
    
    extract_email = udf(lambda cell: str(next(filter(lambda t: t["name"] == "Email", cell), {}).get("value", "")), StringType())
    extract_mobile = udf(lambda cell: str(next(filter(lambda t: t["name"] == "Mobile", cell), {}).get("value", "")), StringType())
    df = df.withColumn('email', extract_email(col("contactMethods"))).withColumn('mobile', extract_mobile(col("contactMethods")))
    display(df)

     

    Profies, please advise!