Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the package. I have the same configuration in an azure synapse notebook and it works perfectly. The interesting thing is that this does work in Fabric if I read the xml file using scala instead.

 

I just tried this on the new 2.2 runtime as well and no luck.

 

Code:

df = spark.read.format("xml").option("rowTag", "BillOfLading").load("Files/Freight/kls/raw/KACC20230724.xml")
 
Error: 
Py4JJavaError: An error occurred while calling o5568.load. : org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: xml. Please find packages at `https://spark.apache.org/third-party-projects.html`.
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers

16 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Joshrodgers123 ,

    Thanks for using Fabric Community.

    Apologies for the issue you have been facing. 

    We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

    Appreciate your patience.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi Joshrodgers123 

      Could you please try to upload the .jar file in library management, and install it then use in notebook?

      Please upload the .jar file here and try running the pyspark code.
      Hope this helps. Please let us know if you have any further questions.

      • ramonsuarez's avatar
        ramonsuarez
        Advocate II
        2 years ago

        Can you provide a link to the .jar file or to the webpage where we can download it please? 

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi  Joshrodgers123 

      Can you please share your workspace id, artifact id ? We'd like to check if it's our issue or hit the error by design. It will be great if you can also share the code snippet along with it. We would like to understand why there is an issue?

      You can send us this information through email to AzCommunity[at]Microsoft[dot]com with the below details,

      Email subject: <Attn - v-nikhilan-msft  :Spark XML does not work with pyspark>

       

      Thanks.

       

      • Joshrodgers123's avatar
        Joshrodgers123
        Advocate V
        2 years ago

        Hi Anonymous, I have emailed all of the requested details. Thanks.

  • Fgarcia1986's avatar
    Fgarcia1986
    Regular Visitor
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers