Forum Discussion

Broeks's avatar
Broeks
Icon for Helper II rankHelper II
1 year ago
Solved

Select the fabric kernel to debug spark job definition in VSCode

Hello,   I want to develop and debug spark job definitions on my local machine using VSCode. I want to use mssparkutils and notebookutils. So I'm wondering how i can select the fabric kernel in VSC...
data engineering
Spark Job Definition
  • Apikpo09's avatar
    Apikpo09
    1 year ago

    Currently, Spark Job Definitions cannot be debugged using the Fabric kernel in VSCode. The Fabric kernel is mainly designed for running notebooks, whereas Spark Job Definitions require execution on the remote Spark Runtime. If you try to debug a Spark Job Definition using the Fabric kernel, it won’t function as expected since it lacks the necessary Spark execution environment.

    To properly develop and debug Spark jobs, you need to select the remote Spark Runtime, which provides the required Spark components and dependencies. This ensures that your job runs in an environment similar to the one it will be deployed in, allowing for accurate testing and debugging.

    If you were expecting this functionality within the Fabric kernel itself, it would be worth checking Microsoft’s official roadmap or documentation for any future support. Having the ability to debug Spark Job Definitions directly in the Fabric kernel would be a valuable enhancement, as it would streamline development and testing workflows without requiring an external runtime.

    Are there any specific issues you’re facing while debugging your Spark Job Definition? Let me know how I can assist further!

v-pnaroju-msft's avatar
v-pnaroju-msft
Icon for Community Support rankCommunity Support
1 year ago

Hi Broeks,

Thank you for reaching out with your query. Kindly follow the steps below to address the issue:

  1. Ensure that the latest Synapse VS Code extension is installed. Verify that OpenJDK 8 is correctly installed and added to the system PATH. Additionally, select the appropriate Fabric runtime (ex:fabric-synapse-runtime-1.2) in VS Code.

  2. Currently, VS Code does not support the Fabric kernel for debugging Spark job definitions. Spark job definitions run locally and do not connect to the Fabric Spark cluster in the cloud. Tools such as mssparkutils and notebookutils are specifically designed for use in Notebooks and Fabric-managed environments, and they are not compatible with local Spark job definitions.

  3. As an alternative, you may develop and debug the code in a Fabric-enabled notebook in VS Code, where mssparkutils and notebookutils are accessible. Once the code is finalized, test the Spark job definition in the Fabric portal to validate its behavior in the cloud.

For further reference, please refer to the following documentation:

If you find this response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members facing similar queries.

 

Thank you.