topic Re: Installing Packages in Data Engineering

Installing Packages

versydney — Sun, 28 May 2023 20:00:25 GMT

Has anyone been able to successfully install the spark-xml library on the spark pool in Fabric?

I added this in the workspace settings:

After a few minutes, it was 'successful', but the notebook still complains:

Am I missing something? Thanks in advance Fabric community! 🙂

Re: Installing Packages

Anonymous — Tue, 30 May 2023 11:49:02 GMT

@versydney - As per the repro, I was experiencing the same issue from my end and reached out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Stay Tuned!

Re: Installing Packages

versydney — Tue, 30 May 2023 19:46:08 GMT

Thanks @Anonymous , looking forward to hearing from MS.

Re: Installing Packages

wahmed20 — Tue, 06 Jun 2023 13:12:14 GMT

You can install packages using the

Re: Installing Packages

versydney — Tue, 06 Jun 2023 21:13:25 GMT

There is no feed library for spark-xml hence I used custom library.

Re: Installing Packages

Joshrodgers123 — Fri, 09 Jun 2023 02:41:34 GMT

@Anonymous Was there any update on this? I'm running into the same issue on my end.

Re: Installing Packages

tpenrose — Fri, 21 Jul 2023 07:42:33 GMT

Has there been any update from Microsoft on this?

I'm running into the same problem as @versydney .

Spark-XML package uploaded via Library Management on the Fabric Workspace:

In my notebook, I have the following code:

And this returns the error:

This is the third iteration of this particular notebook, having first written it a couple of years ago to work on Databricks, then last year it was converted to a Synapse Spark Pool notebook and now it's been converted to a Microsoft Fabric notebook.

In Databricks and Synapse, simply importing the spark-xml package was enough to get it working, but for some reason the same approach isn't working in Fabric.

Re: Installing Packages

Joshrodgers123 — Fri, 21 Jul 2023 15:19:35 GMT

I opened a support ticket for this, but they never got it working and said it was not supported now. They didn't give any timeline or any good reason as to why it doesn't work.

As a workaround, I am reading the XML with scala, which does work with this package. Then just doing the rest of what I need to do in Pysypark.

Re: Installing Packages

tpenrose — Sun, 23 Jul 2023 11:39:05 GMT

Thanks @Joshrodgers123, that's a great suggestion.

With a little help from ChatGPT I was able to rewrite parts of my notebook to use Scala, and then drop back into Pyspark to do the rest of the work.

It's a shame that Microsoft haven't got this working properly, especially as it works in Synapse Analytics, but this workaround will do for now.

Re: Installing Packages

koenverbeeck — Thu, 12 Oct 2023 19:43:39 GMT

I'm having the same issue with spark-xml. Have there been any updates on this?

Re: Installing Packages

Joshrodgers123 — Thu, 12 Oct 2023 19:47:15 GMT

@Anonymous Can you let us know if there is any feedback from the product team as to why this does not work and if there is a planned ETA for when this will work? This works perfectly fine in Synapse notebooks.