Graphframes error: ClassNotFoundException

[goreavin]

I am trying to use Graphframes library to generate GraphX objects.

Output of "%conda list" shows : graphframes 0.6 pyhd3deb0d_1 conda-forge

So the package is installed. But when I try to construct a GraphFrame

g = GraphFrame(v,edge)

I get the following error:
Py4JJavaError: An error occurred while calling o5762.loadClass. : java.lang.ClassNotFoundException: org.graphframes.GraphFramePythonAPI

What do I need to get this to work?

[goreavin · Accepted solution]

@Anonymous : I received a message from your support engineer with the following solutions:

1. Use Scala instead of Python:
2. Add the library manually to the environment:
   • Install the jar from maven repository. reference: Maven Repository: graphframes » graphframes » 0.8.0-spark3.0-s_2.12 (mvnrepository.com)
   • upload the Jar to your environment. Reference: Library management in Fabric environments - Microsoft Fabric | Microsoft Learn
   • Attach the environment to your workspace. Reference: Create, configure, and use an environment in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
   • Add this config as first cell inside your notebook:
     

%%configure -f
{    
    "conf": {
        "spark.jars.packages": "graphframes:graphframes:0.8.0-spark3.0-s_2.12"
    }
}

This solved the issue. 

Thanks! Avi

[_maclura_]

@goreavin your suggested solution does work for me in a Fabric Notebook even using Python.

My custom environment uses Spark 3.5 and Scala 2.12, so:
- I downloaded graphframes-0.8.4-spark3.5-s_2.12.jar from https://spark-packages.org/package/graphframes/graphframes
- I imported it as Custom library in my custom environment. 
- I attached this custom environment to my notebook workspace. 
- I added as a first cell of my notebook:

%%configure
{
    "conf": {
        "spark.jars.packages": "graphframes:graphframes:0.8.4-spark3.5-s_2.12"
    }
}

- I added at the beginning of my PySpark cell:

from graphframes import GraphFrame
spark.sparkContext.setCheckpointDir("Files/graphframes_checkpoints")

- and then I started working with graphframes in PySpark.

Note: setCheckpointDir  is required by GraphFrame when calculating connected components. And the "Files/graphframes_checkpoints" path is relative to the default lakehouse path, so you need at least a default lakehouse attached to your notebook.

I can now survive in Fabric even without SQL Server's recursive CTE support. 😉

[Anonymous]

Hi @goreavin ,

Glad to know that your issue got resolved.
Please continue using Fabric Comunity on your further queries.

[Anonymous]

Hi @goreavin ,

Thanks for using Fabric Community. I apologize for the issue you are facing.

These are the few steps to resolve your issue:

1. Include GraphFrames in Spark Session Configuration:

Even though graphframes is installed in your conda environment, it might not be included in the Spark session. You need to explicitly tell Spark to use it. Here's how:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
  .master("local[*]") \
  .config("spark.jars.packages", "graphframes:graphframes:0.6") \
  .getOrCreate()

# Now you can create your GraphFrame
g = GraphFrame(v, edge)

Explanation: 
spark.jars.packages: This configuration tells Spark to include the JAR file for graphframes (version 0.6 in this case) when launching the Spark application.

2. Verify Spark Version Compatibility:

Double-check the compatibility between your Spark version and the installed graphframes version (0.6). Refer to the GraphFrames documentation for supported Spark versions for your specific graphframes version - link. There might be a mismatch causing the class not to be found.

3. Reinstall or Update GraphFrames (if necessary):

If the above steps don't work, consider reinstalling graphframes.

By following these steps, you should be able to resolve the ClassNotFoundException.

Hope this helps you in resolving the issue. Please let me know incase if you have further queries.

[Anonymous]

Hi @goreavin ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
Otherwise, will respond back with the more details and we will try to help .

[Anonymous]

Hi @goreavin ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
Otherwise, will respond back with the more details and we will try to help .