Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Icon for Advocate V rankAdvocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Joshrodgers123's avatar
Joshrodgers123
Icon for Advocate V rankAdvocate V
2 years ago

Hi Anonymous, I have emailed all of the requested details. Thanks.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Joshrodgers123 ,
Thanks for providing the information. I have given the details to the internal team. I will update you once I hear back from them.
Appreciate your patience.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi  Joshrodgers123 

    To use PySpark in order to play with XML files, we have to use spark-xml package Link1

    Try using the Scala API

    %%spark

    val df = spark.read
                    .format("com.databricks.spark.xml")
                    .option("rowTag", "book")
                    .load("file:///synfs/nb_resource/builtin/demo.xml")

    df.show(10)

     

     You can find the tutorial here: Link2

     

    So basically, the format must be .format("com.databricks.spark.xml").

     

     

    • Joshrodgers123's avatar
      Joshrodgers123
      Icon for Advocate V rankAdvocate V
      2 years ago

      I have already installed that package. The code you provided is scala, which does work. Pyspark does not work though. 

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi Joshrodgers123 
        Apologies for the delay in response.

        I would request you to please go ahead with Microsoft support for this. Please raise a support ticket on this link: https://support.fabric.microsoft.com/en-US/support/.

        Also once you have opened the support ticket , please do share the supportcase# here so that we can keep an eye on it.

        Thanks