Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago
Solved

Spark XML does not work with pyspark

Has anyone been able to read XML files in a notebook using pyspark yet? I loaded the spark-xml_2.12-0.16.0.jar library and am trying to run the below code, but it does not seem to recognize the packa...
data engineering
  • Fgarcia1986's avatar
    Fgarcia1986
    2 years ago

     

    One way that i found is:

    1 - Create an enviroment

    2 - upload the the file spark-xml_2.12-0.17.0.jar

     

     

     

    Open your notebook and language choose spark(scala) and then place the code below:

    %%configure -f
    {"conf": {"spark.jars.packages": "com.databricks:spark-xml_2.12:0.16.0"}}

     

    IMPORTANT: Must be the first code in the session and you can use the environment WorkSpace Default, you don´t have to use the environment that you´ve created, i don´t know but worked.
     
    Then you can change your language to PySpark(Python) and read xml
     
    It takes from 2 to 3 minutes to execute.
     
    Let me know if you have any doubt.
     
    I hope works for everyone
     
    Cheers
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Joshrodgers123 
Apologies for the delay in response.

I would request you to please go ahead with Microsoft support for this. Please raise a support ticket on this link: https://support.fabric.microsoft.com/en-US/support/.

Also once you have opened the support ticket , please do share the supportcase# here so that we can keep an eye on it.

Thanks

Joshrodgers123's avatar
Joshrodgers123
Advocate V
2 years ago

Here is the support ticket: 2311150040007106

  • Joshrodgers123's avatar
    Joshrodgers123
    Advocate V
    2 years ago

    It doesn't seem to be supported with pyspark. I got it working by loading the data with scala and then doing my transformations with pyspark. 

  • ramonsuarez's avatar
    ramonsuarez
    Advocate II
    2 years ago

    My workaround is loading into a Pandas dataframe and then converting it to a pyspark dataframe before writing to delta tables.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Joshrodgers123 
    Thanks for the details. We expect you to keep using this forum and also motivate others to do that same. 
    Thanks

  • ramonsuarez's avatar
    ramonsuarez
    Advocate II
    2 years ago

    Josh Did you get a reply on how to do use spark-xml with pyspark in Fabric? Thanks