Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Icon for Advocate V rankAdvocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Joshrodgers123's avatar
Joshrodgers123
Icon for Advocate V rankAdvocate V
2 years ago

That is where I have been loading it. 

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi  Joshrodgers123 

    Can you please share your workspace id, artifact id ? We'd like to check if it's our issue or hit the error by design. It will be great if you can also share the code snippet along with it. We would like to understand why there is an issue?

    You can send us this information through email to AzCommunity[at]Microsoft[dot]com with the below details,

    Email subject: <Attn - v-nikhilan-msft  :Spark XML does not work with pyspark>

     

    Thanks.

     

    • Joshrodgers123's avatar
      Joshrodgers123
      Icon for Advocate V rankAdvocate V
      2 years ago

      Hi Anonymous, I have emailed all of the requested details. Thanks.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi Joshrodgers123 ,
        Thanks for providing the information. I have given the details to the internal team. I will update you once I hear back from them.
        Appreciate your patience.