How to use java/scala udf in pyspark?

[gregahren]

I created my java udf in jar which I uploaded in Fabric environment in Custom Libraries section. When I try to run this piece of code in Fabric notebook:

spark.udf.registerJavaFunction("string_lenght", "test.org.apache.spark.sql.JavaStringLength")

The error I get is this:

My question is how can someone take advantage of the fast udf functions written in java/scala when in fabric environment, and how to register them to classpath in pyspark context? 

[gregahren · Accepted solution]

I contacted the support and got back from them.

You can register java UDF from your spark notebook in fabric in one of two ways:

Of course the first thing to do is to include jar in your Fabric Environment in Custom Libraries section.

1. Then you can register UDF in the following way. Switch from pyspark in the cell to Scala.

2. The second solution is to use magic configure with pyspark. But I think this only work in the first cell of the notebook.

Extra Solution: Also I discovered another workaroud which works with pure python UDF registration also:

In Fabric Environment we set up additional Spark Property:

spark.jars -> abfss://...@onelake.dfs.fabric.microsoft.com/.../Files/packages/spark-simple-1.0-SNAPSHOT.jar

That seems to include jar file without specifing it in custom libraries section, and also provides SparkContext JVM with jars namespaces objects to be available to use in python code inside of the notebook.

```python

sc._jvm.my.java.package.spark.simple.Register()

```

It would be nice that this is automatically set up by Fabric environment custom jars that user uploads. So it is not needed to provide extra property setup.

[v-nikhilan-msft]

Hi @gregahren 
Thanks for providing the resolution here. Please continue using Fabric Community for any help regarding your queries.

[v-nikhilan-msft]

Hi @gregahren 
Thanks for using Fabric Community.

This might require a deeper investigation from our engineering team to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Thanks.

[v-nikhilan-msft]

Hi @gregahren 
We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please provide the details here.
Otherwise, will respond back with the more details and we will try to help.
Thanks

[gregahren]

Hi,

@v-nikhilan-msft I am still in progress of talking to support. However I will update the thread with the solution once we close the ticket. 

thanks