Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Icon for Advocate V rankAdvocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Joshrodgers123's avatar
Joshrodgers123
Icon for Advocate V rankAdvocate V
2 years ago

It doesn't seem to be supported with pyspark. I got it working by loading the data with scala and then doing my transformations with pyspark. 

ramonsuarez's avatar
ramonsuarez
Icon for Advocate II rankAdvocate II
2 years ago

My workaround is loading into a Pandas dataframe and then converting it to a pyspark dataframe before writing to delta tables.