Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Fgarcia1986's avatar
Fgarcia1986
Regular Visitor
2 years ago

 

One way that i found is:

1 - Create an enviroment

2 - upload the the file spark-xml_2.12-0.17.0.jar

 

 

 

Open your notebook and language choose spark(scala) and then place the code below:

%%configure -f
{"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

 

IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
 
Then you can change your language to PySpark(Python) and read xml
 
It takes from 2 to 3 minutes to execute.
 
Let me know if you have any doubt.
 
I hope works for everyone
 
Cheers