Forum Discussion
Helper IISelect the fabric kernel to debug spark job definition in VSCode
Currently, Spark Job Definitions cannot be debugged using the Fabric kernel in VSCode. The Fabric kernel is mainly designed for running notebooks, whereas Spark Job Definitions require execution on the remote Spark Runtime. If you try to debug a Spark Job Definition using the Fabric kernel, it won’t function as expected since it lacks the necessary Spark execution environment.
To properly develop and debug Spark jobs, you need to select the remote Spark Runtime, which provides the required Spark components and dependencies. This ensures that your job runs in an environment similar to the one it will be deployed in, allowing for accurate testing and debugging.
If you were expecting this functionality within the Fabric kernel itself, it would be worth checking Microsoft’s official roadmap or documentation for any future support. Having the ability to debug Spark Job Definitions directly in the Fabric kernel would be a valuable enhancement, as it would streamline development and testing workflows without requiring an external runtime.
Are there any specific issues you’re facing while debugging your Spark Job Definition? Let me know how I can assist further!
Currently, Spark Job Definitions cannot be debugged using the Fabric kernel in VSCode. The Fabric kernel is mainly designed for running notebooks, whereas Spark Job Definitions require execution on the remote Spark Runtime. If you try to debug a Spark Job Definition using the Fabric kernel, it won’t function as expected since it lacks the necessary Spark execution environment.
To properly develop and debug Spark jobs, you need to select the remote Spark Runtime, which provides the required Spark components and dependencies. This ensures that your job runs in an environment similar to the one it will be deployed in, allowing for accurate testing and debugging.
If you were expecting this functionality within the Fabric kernel itself, it would be worth checking Microsoft’s official roadmap or documentation for any future support. Having the ability to debug Spark Job Definitions directly in the Fabric kernel would be a valuable enhancement, as it would streamline development and testing workflows without requiring an external runtime.
Are there any specific issues you’re facing while debugging your Spark Job Definition? Let me know how I can assist further!
BroeksHello Apikpo09,
Thanks for your response!
Not the answer I was hoping for, but this explains why I couldn't select the fabric kernel.
Is there a way for me to request this as a new feature? As it would enable us to develop a maintainable and modulair codebase.
I want to develop a modulair codebase to facilitate our ETL process. As this is currently developed in multiple notebooks, which aren't very usefull when developing/debugging functions/classes accross multiple notebooks. Also the merging notebooks is a but of a hassle (putting it mildly :P).
However with the conda runtime I am able to develop/debug against a spark enviroment.
But specific issues I'm facing are:
- Wanting to use the latest spark/delta features, which are available in spark runtime 1-3
- Wanting to use notebookutils within my functions/classes.