Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Joshrodgers123 

Could you please try to upload the .jar file in library management, and install it then use in notebook?

Please upload the .jar file here and try running the pyspark code.
Hope this helps. Please let us know if you have any further questions.

ramonsuarez's avatar
ramonsuarez
Advocate II
2 years ago

Can you provide a link to the .jar file or to the webpage where we can download it please?