Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
gregahren
Frequent Visitor

How to use java/scala udf in pyspark?

I created my java udf in jar which I uploaded in Fabric environment in Custom Libraries section. When I try to run this piece of code in Fabric notebook:

 

 

spark.udf.registerJavaFunction("string_lenght", "test.org.apache.spark.sql.JavaStringLength")

 

 

The error I get is this:

gregahren_0-1704377799360.png

My question is how can someone take advantage of the fast udf functions written in java/scala when in fabric environment, and how to register them to classpath in pyspark context? 

1 ACCEPTED SOLUTION
gregahren
Frequent Visitor

I contacted the support and got back from them.

You can register java UDF from your spark notebook in fabric in one of two ways:

Of course the first thing to do is to include jar in your Fabric Environment in Custom Libraries section.

 

1. Then you can register UDF in the following way. Switch from pyspark in the cell to Scala.

gregahren_0-1704891604810.png
2. The second solution is to use magic configure with pyspark. But I think this only work in the first cell of the notebook.

gregahren_1-1704891604812.png

 

Extra Solution: Also I discovered another workaroud which works with pure python UDF registration also:

In Fabric Environment we set up additional Spark Property:

spark.jars -> abfss://...@onelake.dfs.fabric.microsoft.com/.../Files/packages/spark-simple-1.0-SNAPSHOT.jar

gregahren_2-1704892031602.png

That seems to include jar file without specifing it in custom libraries section, and also provides SparkContext JVM with jars namespaces objects to be available to use in python code inside of the notebook.

```python

sc._jvm.my.java.package.spark.simple.Register()
```

It would be nice that this is automatically set up by Fabric environment custom jars that user uploads. So it is not needed to provide extra property setup.

 

View solution in original post

5 REPLIES 5
gregahren
Frequent Visitor

I contacted the support and got back from them.

You can register java UDF from your spark notebook in fabric in one of two ways:

Of course the first thing to do is to include jar in your Fabric Environment in Custom Libraries section.

 

1. Then you can register UDF in the following way. Switch from pyspark in the cell to Scala.

gregahren_0-1704891604810.png
2. The second solution is to use magic configure with pyspark. But I think this only work in the first cell of the notebook.

gregahren_1-1704891604812.png

 

Extra Solution: Also I discovered another workaroud which works with pure python UDF registration also:

In Fabric Environment we set up additional Spark Property:

spark.jars -> abfss://...@onelake.dfs.fabric.microsoft.com/.../Files/packages/spark-simple-1.0-SNAPSHOT.jar

gregahren_2-1704892031602.png

That seems to include jar file without specifing it in custom libraries section, and also provides SparkContext JVM with jars namespaces objects to be available to use in python code inside of the notebook.

```python

sc._jvm.my.java.package.spark.simple.Register()
```

It would be nice that this is automatically set up by Fabric environment custom jars that user uploads. So it is not needed to provide extra property setup.

 

Hi @gregahren 
Thanks for providing the resolution here. Please continue using Fabric Community for any help regarding your queries.

v-nikhilan-msft
Community Support
Community Support

Hi @gregahren 
Thanks for using Fabric Community.

This might require a deeper investigation from our engineering team to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Thanks.

Hi @gregahren 
We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please provide the details here.
Otherwise, will respond back with the more details and we will try to help.
Thanks

Hi,

 

@v-nikhilan-msft I am still in progress of talking to support. However I will update the thread with the solution once we close the ticket. 

thanks

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Expanding the Synapse Forums

New forum boards available in Synapse

Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

Top Solution Authors
Top Kudoed Authors