Forum Discussion
Helper IISelect the fabric kernel to debug spark job definition in VSCode
Currently, Spark Job Definitions cannot be debugged using the Fabric kernel in VSCode. The Fabric kernel is mainly designed for running notebooks, whereas Spark Job Definitions require execution on the remote Spark Runtime. If you try to debug a Spark Job Definition using the Fabric kernel, it won’t function as expected since it lacks the necessary Spark execution environment.
To properly develop and debug Spark jobs, you need to select the remote Spark Runtime, which provides the required Spark components and dependencies. This ensures that your job runs in an environment similar to the one it will be deployed in, allowing for accurate testing and debugging.
If you were expecting this functionality within the Fabric kernel itself, it would be worth checking Microsoft’s official roadmap or documentation for any future support. Having the ability to debug Spark Job Definitions directly in the Fabric kernel would be a valuable enhancement, as it would streamline development and testing workflows without requiring an external runtime.
Are there any specific issues you’re facing while debugging your Spark Job Definition? Let me know how I can assist further!
Helper IIThanks for your response.
I think I've got my VsCode configured correctly. As I'm able to select the microsoft fabric runtime within a notebook in VSCode. However, I'm not able to do the same in VSCode for a spark job definition.
- nilendraFabric
Super User
Can you please confirm that the Java version which you are using is OpenJDK 8
Select the right spark runtime version. You can set environment in notebook in the portal, if you use fabric spark 1.2, you need to choose fabric-synapse-runtime-1-2 kernel when running code in VS Code
Hope this helps
- Broeks
I've got the openjdk8 installed and in the path. It's pointing to the path for the openjdk: "C:\Program Files\Eclipse Adoptium\jre-8.0.442.6-hotspot\bin"
When I get the java version it says:So i would assume it's correct.
When running the pyspark code in vscode I get the following error for the fabric 1-2 runtime:
And it also leaves me with the other question. Can I debug my spark job defintion code against the fabric spark cluster in the cloud. As I understand it correctly I'm running against a local version of spark and cannot use for example mssparkutils? Of fabric runtime 1-3?
