Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi  Joshrodgers123 

Can you please share your workspace id, artifact id ? We'd like to check if it's our issue or hit the error by design. It will be great if you can also share the code snippet along with it. We would like to understand why there is an issue?

You can send us this information through email to AzCommunity[at]Microsoft[dot]com with the below details,

Email subject: <Attn - v-nikhilan-msft  :Spark XML does not work with pyspark>

 

Thanks.

 

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago

Hi Anonymous, I have emailed all of the requested details. Thanks.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Joshrodgers123 ,
    Thanks for providing the information. I have given the details to the internal team. I will update you once I hear back from them.
    Appreciate your patience.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi  Joshrodgers123 

      To use PySpark in order to play with XML files, we have to use spark-xml package Link1

      Try using the Scala API

      %%spark

      val df = spark.read
                      .format("com.databricks.spark.xml")
                      .option("rowTag", "book")
                      .load("file:///synfs/nb_resource/builtin/demo.xml")

      df.show(10)

       

       You can find the tutorial here: Link2

       

      So basically, the format must be .format("com.databricks.spark.xml").

       

       

      • Joshrodgers123's avatar
        Joshrodgers123
        Advocate V
        2 years ago

        I have already installed that package. The code you provided is scala, which does work. Pyspark does not work though.