Forum Discussion
Helper IIDebugging in VSCode: pyspark.errors.exceptions.captured.IllegalArgumentException
Hi Broeks,
Thank you for your follow-up. It appears that you have diligently troubleshot and configured the devcontainer for Spark Job Definition (SJD) development. Based on my understanding, the approach you have adopted regarding base image selection, required extensions, and authentication settings is appropriate.
- Using version v2.1.0 for Base Image is a suitable choice, as it incorporates the latest updates and fixes.
- Including the Azure Account and Azure Resources extensions for VS Code Extensions is a prudent decision. These are essential for authentication, even though they may not always be part of example repositories.
- For authentication settings, ensuring proper token refresh is considered a best practice for maintaining seamless authentication.
If you continue to encounter any issues, we kindly suggest reaching out to the Microsoft Support Team through the link provided below. Raising a support ticket will help you receive tailored insights related to your account and potential resolutions.
Microsoft Fabric Support and Status | Microsoft Fabric
If you find our response helpful, we kindly request you to mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.
Thank you.
Community SupportHi Broeks,
Thank you for your valuable update. We kindly request you to keep us informed about the issue, as it may be beneficial for other community members facing similar concerns.
Thank you.
BroeksYes, will do!
One of the first fixed I got from the support team is on using the correct fabric runtime in both your local and remote SJD. So in my case I was using the runtime 1-2 locally, but my remote SJD was configured to the workspace default, which was configured to runtime 1-3.
So first fix for an error like this:ERROR] 2025-04-01 09:33:45.448 [lighter-poll-status] NotebookSessionClient: {"code":"InternalError","subCode":0,"message":"An internal error occurred.","timeStamp":"2025-04-01T09:33:45.9960955Z","httpStatusCode":500,"hresult":-2147467259,"details":[{"code":"RootActivityId","message":"103d2651-6b75-41c7-87e2-5e8246c5c633"},{"code":"Param1","message":"Response status code does not indicate success: 430 ()."}]} [ERROR] 2025-04-01 09:33:45.449 [lighter-poll-status] LighterClientState: fetch status java.lang.IllegalArgumentException: Can not create session for lighter. {"code":"InternalError","subCode":0,"message":"An internal error occurred.","timeStamp":"2025-04-01T09:33:45.9960955Z","httpStatusCode":500,"hresult":-2147467259,"details":[{"code":"RootActivityId","message":"103d2651-6b75-41c7-87e2-5e8246c5c633"},{"code":"Param1","message":"Response status code does not indicate success: 430 ()."}]} at org.apache.spark.lighter.client.auth.datacloud.NotebookSessionClient.createSession(NotebookSessionClient.scala:137) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.auth.DataCloudProvider.init(DataCloudProvider.scala:140) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.auth.DataCloudProvider.<init>(DataCloudProvider.scala:110) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.auth.EndpointConfigProviderBuilder.build(EndpointConfigProviderBuilder.scala:12) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientContext.init(LighterClientContext.scala:57) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientContext.<init>(LighterClientContext.scala:45) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientContext$.context$lzycompute(LighterClientContext.scala:123) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientContext$.context(LighterClientContext.scala:123) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientContext$.getOrCreate(LighterClientContext.scala:125) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientState$.fetchStatus(LighterClientState.scala:99) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.lighter.client.LighterClientState$.$anonfun$new$1(LighterClientState.scala:80) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1454) ~[spark-core_2.12-3.4.1.jar:3.4.1] at org.apache.spark.lighter.client.LighterClientState$$anon$1.run(LighterClientState.scala:90) ~[spark-lighter-core_2.12-2.0.11_spark-3.4.0.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_372] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_372] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_372] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_372] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]