topic Re: Spark Lib classpath not set - similar type of issue reported in 1085 in Data Engineering

Spark Lib classpath not set - similar type of issue reported in 1085

saravrathinam — Thu, 23 Oct 2025 23:40:51 GMT

I am writing to seek your assistance regarding an issue with setting the Spark classpath for the Redshift JDBC driver (redshift-jdbc42-2.2.0).

Here are the steps I have followed so far:

I created an Environment, named it RedshiftLib, and then navigated to Custom Library. Here, I uploaded the file (redshift-jdbc42-2.2.0.jar), saved it, and published the changes.
After publishing, I created a new Notebook, attached the Environment to it, and started a new session.
In the Notebook, I executed the following command to validate whether the classpath was set:

```python

print("spark.jars =", spark.sparkContext.getConf().get("spark.jars",""))

cp = spark._jvm.java.lang.System.getProperty("java.class.path")

print("Found 'redshift' on classpath? ", "redshift" in cp.lower())

print(cp[:1200])

```

The result I received was:

```python

spark.jars =

Found 'redshift' on classpath? False

com.amazon.redshift.Driver (2.x): False

```

Alternatively, I loaded the file into the ABFS path and tried to register the path with the following command, but it still didn’t work:

```python

%%configure -f

{

"jars": [

"abfss://xxxxxxxx@onelake.dfs.fabric.microsoft.com/xxxxxx.Lakehouse/Files/Common/Library/redshift-jdbc42-2.2.0.jar"

]

}

```

After connecting the Environment in the Notebook, I removed the existing published library (redshift-jdbc42-2.2.0.jar), reloaded the same file, published it again, and tried to execute the validation command once more. Unfortunately, I am still encountering the same error—the classpath is not set.

This issue seems to be an extension of the known bug 1085. Could you kindly provide some inputs or guidance on how to resolve this problem?

Thank you for your time and support.

Re: Spark Lib classpath not set - similar type of issue reported in 1085

Anonymous — Fri, 24 Oct 2025 15:27:39 GMT

Hi @saravrathinam ,

Thanks for reaching out to the Microsoft fabric community forum.

Please follow this steps which may help you resolve your issue:

Upload your JAR to the Environment as described above (via the “Custom Library” section). Wait until publish completes.
In the notebook, stop your current session and completely restart it (so Spark picks up the library at session start).
Ensure the notebook is attached to the correct Environment (the one with the JAR).
After the session starts, run your validation code to ensure the classpath includes the JAR.
Then attempt a JDBC connection using the driver class name.

Hope this helps. Please feel free to reach out for any further questions.

Thank you.

Re: Spark Lib classpath not set - similar type of issue reported in 1085

Anonymous — Tue, 28 Oct 2025 04:05:50 GMT

Hi @saravrathinam ,

I wanted to check if you had the opportunity to review the information provided . If you still require support, please let us know, we are happy to assist you.

Thank you.

Re: Spark Lib classpath not set - similar type of issue reported in 1085

saravrathinam — Tue, 28 Oct 2025 04:09:02 GMT

Its not working as expected. I did the samething initially too. I have created the a new fresh env without any Custom Lib and followed your steps, still the same issue persistent.

Re: Spark Lib classpath not set - similar type of issue reported in 1085

saravrathinam — Wed, 29 Oct 2025 08:48:08 GMT

I'm encountering the same error with integrating JBBC Redshift connector (jar file) into a new environment. Despite multiple attempts, I've been unable to resolve the problem.

Here's a detailed breakdown of what I've done so far: I created a new environment and added jar file into the Custom Library, which I downloaded and uploaded to the Custom Library. After starting a new session, I attached the notebook session to the newly created environment and ran the following code:

```python

print("spark.jars =", spark.sparkContext.getConf().get("spark.jars",""))

cp = spark._jvm.java.lang.System.getProperty("java.class.path")

print("Found 'redshift' on classpath? ", "redshift" in cp.lower())

print(cp[:1200]) # peek

def has_class(cls):

try:

spark._jvm.java.lang.Class.forName(cls); return True

except Exception: return False

print("com.amazon.redshift.Driver (2.x):", has_class("com.amazon.redshift.Driver"))

```

The result I received was:

```

spark.jars =

Found 'redshift' on classpath? False

```

I'm not sure how to proceed from here. It appears that the Redshift driver is not being recognized on the classpath. Could anyone provide some guidance on how to resolve this issue? Any insights or suggestions would be greatly appreciated.

Additional information:

in the spark setting in the workspace setting
runtime version
1.3 (spark 3.5, delta 3.2)
will it useful to debug ?

Re: Spark Lib classpath not set - similar type of issue reported in 1085

Anonymous — Mon, 03 Nov 2025 05:50:43 GMT

Hi @saravrathinam ,

As you have already completed all the suggested troubleshooting steps and the issue persists, I recommend raising a support ticket with Microsoft. This should assist in resolving the problem.
Please use the below link for raising the support ticket : Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

Thank you.

Re: Spark Lib classpath not set - similar type of issue reported in 1085

Anonymous — Fri, 07 Nov 2025 08:52:29 GMT

Hi @saravrathinam ,

Could you please confirm if the issue has been resolved through the support ticket with Microsoft? If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community.

Please feel free to contact us if you have any further questions.

Thank you.