How to use java/scala udf in pyspark?

I contacted the support and got back from them.

You can register java UDF from your spark notebook in fabric in one of two ways:

Of course the first thing to do is to include jar in your Fabric Environment in Custom Libraries section.

1. Then you can register UDF in the following way. Switch from pyspark in the cell to Scala.

2. The second solution is to use magic configure with pyspark. But I think this only work in the first cell of the notebook.

Extra Solution: Also I discovered another workaroud which works with pure python UDF registration also:

In Fabric Environment we set up additional Spark Property:

spark.jars -> abfss://[email protected]/.../Files/packages/spark-simple-1.0-SNAPSHOT.jar

That seems to include jar file without specifing it in custom libraries section, and also provides SparkContext JVM with jars namespaces objects to be available to use in python code inside of the notebook.

```python

It would be nice that this is automatically set up by Fabric environment custom jars that user uploads. So it is not needed to provide extra property setup.