Debugging in VSCode: pyspark.errors.exceptions.captured.IllegalArgumentException

[Broeks]

Hello,

I'm running into another issue with my devcontainer fabric data enginering extension. I'm trying to run a spark job defintion locally on the conda spark runtime 1.2. When I run the code I'm getting this error:

The error doesn't tell me anything, but the code does work in the fabric SJD (without the config settings ofcourse). Can you help me to fix this error? I'm even getting this error when I'm just creating an dataframe defined in code, so it's not linked to remote storage. Or even beter, help me with the following more general question.

A more general question:

As you may have noticed I'm trying to get the devopment of SJD in VSCode devcontainers to work. But I'm constantly running into issue's. The support on this forum is great, but it still feels buggy to get this to work. I'm using the following documentation as my guide lines, but it's not very substansial. 
VS Code extension overview - Microsoft Fabric | Microsoft Learn
VS Code extension with Docker support - Microsoft Fabric | Microsoft Learn

Issues that lead me to change the default repository code are: 
- Latest docker image when building my containerimage from 2.06 tot 2.1.0:

FROM mcr.microsoft.com/msfabric/synapsevscode/fabric-synapse-vscode:v2.1.0

- In my dockerfile: Change the pyspark installer from the default conda installer to pip

# Install PySpark in the base Conda environment
ENV SPARK_VERSION=3.4.1
RUN pip install pyspark==$SPARK_VERSION

I've also used mamba for this

# Install mamba and PySpark
ENV SPARK_VERSION=3.4.1
RUN conda install -n base -c conda-forge mamba -y && \
    mamba install -n base -c conda-forge pyspark==$SPARK_VERSION -y && \
    conda clean --all -y

As the original conda solver had issue with resolving dependency's and was not able to build an image.

The default checks I'm doing are:
What's my JAVA_HOME?

Am I on the latest (prerelease)version of the fabric data enginering extension
 

Then I've ran into issue's with:
- A refresh token: Solved: Re: Debugging in VSCode: Failed to get refresh tok... - Microsoft Fabric Community
-- which was fixed by adding the Azure Resources extension to my devcontainer file.
- Trident-Spark authentication: Solved: Re: Trident-Spark-IDE authentication with each spa... - Microsoft Fabric Community
-- Which was fixed by a newer version of the fabric data enginering extension
- I've also had the spark driver
-- Which was fixed by setting the spark driver to local host in my spark config in de main.py:

conf.set("spark.driver.host", "localhost")

-- I don't think this option needs to be set, because it's not in the config you provide, but it does get met a step further in running my code.

All in all, I need some help getting a working dockerimage with devcontainer to start developing spark job definitions with a team. I don't want to be dependend on individial installment processen on individual PC's which can posse configuration issues. Can you help me with a clear an working example of a dockerimage devcontainer combination?

Kind regards,

Martijn Broeks

[v-pnaroju-msft]

Hi Broeks,

Thank you for sharing this valuable insight.

[v-pnaroju-msft]

Hi Broeks,

Thank you for your valuable update. We kindly request you to keep us informed about the issue, as it may be beneficial for other community members facing similar concerns.

Thank you.

[Broeks]

Yes, will do!
One of the first fixed I got from the support team is on using the correct fabric runtime in both your local and remote SJD. So in my case I was using the runtime 1-2 locally, but my remote SJD was configured to the workspace default, which was configured to runtime 1-3.

So first fix for an error like this:

ERROR] 2025-04-01 09:33:45.448 [lighter-poll-status] NotebookSessionClient: {"code":"InternalError","subCode":0,"message":"An internal error occurred.","timeStamp":"2025-04-01T09:33:45.9960955Z","httpStatusCode":500,"hresult":-2147467259,"details":[{"code":"RootActivityId","message":"103d2651-6b75-41c7-87e2-5e8246c5c633"},{"code":"Param1","message":"Response status code does not indicate success: 430 ()."}]}
[ERROR] 2025-04-01 09:33:45.449 [lighter-poll-status] LighterClientState: fetch status
java.lang.IllegalArgumentException: Can not create session for lighter. {"code":"InternalError","subCode":0,"message":"An internal error occurred.","timeStamp":"2025-04-01T09:33:45.9960955Z","httpStatusCode":500,"hresult":-2147467259,"details":[{"code":"RootActivityId","message":"103d2651-6b75-41c7-87e2-5e8246c5c633"},{"code":"Param1","message":"Response status code does not indicate success: 430 ()."}]}
	at org.apache.spark.lighter.client.auth.datacloud.NotebookSessionClient.createSession(NotebookSessionClient.scala:137) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.auth.DataCloudProvider.init(DataCloudProvider.scala:140) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.auth.DataCloudProvider.<init>(DataCloudProvider.scala:110) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.auth.EndpointConfigProviderBuilder.build(EndpointConfigProviderBuilder.scala:12) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientContext.init(LighterClientContext.scala:57) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientContext.<init>(LighterClientContext.scala:45) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientContext$.context$lzycompute(LighterClientContext.scala:123) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientContext$.context(LighterClientContext.scala:123) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientContext$.getOrCreate(LighterClientContext.scala:125) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientState$.fetchStatus(LighterClientState.scala:99) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.lighter.client.LighterClientState$.$anonfun$new$1(LighterClientState.scala:80) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1454) ~[spark-core_2.12-3.4.1.jar:3.4.1]
	at org.apache.spark.lighter.client.LighterClientState$$anon$1.run(LighterClientState.scala:90) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_372]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_372]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_372]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_372]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]

[v-pnaroju-msft]

Hi Broeks,

We are reaching out to follow up on the support ticket you raised. If your issue has been resolved, we would appreciate it if you could share the solution with the community and mark it as the accepted answer. This will help others facing similar challenges and contribute to the collective knowledge of the community.

Thank you.

[Broeks]

Yes, will do.

We are still in contact en they are looking for a solution.

[v-pnaroju-msft]

Hi Broeks,

We are following up to check whether the support ticket raised from your end has provided a resolution to the issue. If it has, we kindly request you to share the solution with the community and mark it as the accepted solution, as this will assist others facing similar challenges and benefit the broader community.

Thank you.

[Broeks]

Yes, will do.

We are still in contact en they are looking for a solution.

[v-pnaroju-msft]

Hi Broeks,

We are following up to check whether you have raised the support ticket. If you have already done so, we kindly request you to share your feedback on the issue raised.

If a solution has been provided, we would appreciate it if you could share it with the community and mark it as the accepted solution. This will help others facing similar challenges and benefit the broader community.

Thank you.

[Broeks]

Hello,

Yes i've raised an issue and will share the solution.

[v-pnaroju-msft · Accepted solution]

Hi Broeks,

Thank you for your follow-up. It appears that you have diligently troubleshot and configured the devcontainer for Spark Job Definition (SJD) development. Based on my understanding, the approach you have adopted regarding base image selection, required extensions, and authentication settings is appropriate.

1. Using version v2.1.0 for Base Image is a suitable choice, as it incorporates the latest updates and fixes.
2. Including the Azure Account and Azure Resources extensions for VS Code Extensions is a prudent decision. These are essential for authentication, even though they may not always be part of example repositories.
3. For authentication settings, ensuring proper token refresh is considered a best practice for maintaining seamless authentication.

If you continue to encounter any issues, we kindly suggest reaching out to the Microsoft Support Team through the link provided below. Raising a support ticket will help you receive tailored insights related to your account and potential resolutions.

Microsoft Fabric Support and Status | Microsoft Fabric

If you find our response helpful, we kindly request you to mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

Thank you.

[Broeks]

Thanks for the suggestion and I will make a support ticket.

I will get back here once I've got a solution!

[Broeks]

Hallo @v-pnaroju-msft,

Thanks for your response.
I've checked my environment variables, as shown in my first post.
I'm assumming the java installation is correct, as I am able to debug with notebooks.

But can you specify the following things a bit more?
For building a stable devcontainer, use the appropriate base image, install the necessary extensions in devcontainer.json, and verify authentication settings.

I'm currently using base image 2.1.0, but have also used 2.0.6
https://mcr.microsoft.com/en-us/artifact/mar/msfabric/synapsevscode/fabric-synapse-vscode/tags

Which extensions do I need? I've noticed that in the microsoft and your github example for exampe there is no Azure Resource extention. And in another issue we've concluded that you need these extensions to authenticate to fabric.
https://github.com/jplane/pyspark-devcontainer/blob/main/.devcontainer/devcontainer.json
https://github.com/microsoft/SynapseVSCode/blob/main/samples/.devcontainer/mariner/Dockerfile

And which authentication settings do I need to verify?

Thanks for your answers!

Martijn Broeks

[v-pnaroju-msft]

Hi @Broeks,

We sincerely appreciate your inquiry on the Microsoft Fabric Community Forum.

Please follow the steps below, which may help resolve the issue:
For Fixing IllegalArgumentException in PySpark:

1. Ensure environmental consistency by using matching versions of Java, PySpark, and dependencies within the devcontainer and Fabric.
2. Use Mamba for dependency resolution instead of pip, as pip may cause conflicts in Conda environments. Additionally, check Spark configurations and ensure that the appropriate settings are defined in the main.py file.
3. Verify the Java setup and ensure that the JAVA_HOME variable is correctly set to OpenJDK 8 within the container.

For building a stable devcontainer, use the appropriate base image, install the necessary extensions in devcontainer.json, and verify authentication settings.

For further reference, kindly visit the following link:
GitHub - jplane/pyspark-devcontainer: A simple VS Code devcontainer setup for local PySpark developm...

If you find our response helpful, we kindly request you to mark it as the accepted solution and provide kudos. This will assist other community members who may face similar queries.

Thank you.