Forum Discussion

goreavin's avatar
goreavin
Helper I
2 years ago
Solved

Graphframes error: ClassNotFoundException

I am trying to use Graphframes library to generate GraphX objects. Output of "%conda list" shows : graphframes 0.6 pyhd3deb0d_1 conda-forge So the package is installed. But when I try to cons...
data engineering
data science
goreavin's avatar
goreavin
Helper I
2 years ago

Anonymous : I received a message from your support engineer with the following solutions:

  1. Use Scala instead of Python:
  2. Add the library manually to the environment:

 

%%configure -f
{    
    "conf": {
        "spark.jars.packages": "graphframes:graphframes:0.8.0-spark3.0-s_2.12"
    }
}

 

 

This solved the issue. 

Thanks! Avi

 

_maclura_'s avatar
_maclura_
Frequent Visitor
5 months ago

goreavin your suggested solution does work for me in a Fabric Notebook even using Python.


My custom environment uses Spark 3.5 and Scala 2.12, so:
- I downloaded graphframes-0.8.4-spark3.5-s_2.12.jar from https://spark-packages.org/package/graphframes/graphframes
- I imported it as Custom library in my custom environment. 
- I attached this custom environment to my notebook workspace. 
- I added as a first cell of my notebook:

%%configure
{
    "conf": {
        "spark.jars.packages": "graphframes:graphframes:0.8.4-spark3.5-s_2.12"
    }
}

- I added at the beginning of my PySpark cell:

from graphframes import GraphFrame
spark.sparkContext.setCheckpointDir("Files/graphframes_checkpoints")

- and then I started working with graphframes in PySpark.

 

Note: setCheckpointDir  is required by GraphFrame when calculating connected components. And the "Files/graphframes_checkpoints" path is relative to the default lakehouse path, so you need at least a default lakehouse attached to your notebook.

 

 

I can now survive in Fabric even without SQL Server's recursive CTE support. 😉