For spark job definitions, are there any plans to allow us to use the Fabric Runtime in VSCode?

[cstoddard]

Hi,

For notebook development using VSCode, you can use the Fabric Runtime to connect to the remote cluster and run code. Is there any timeline to be able to do this with spark job definitions? I would like to move my pipelines to start using SJD's, but it feels like they are a 2nd class citizen compared to notebooks currently.

Thanks!

[v-pnaroju-msft]

Hi cstoddard,

We have not received a response from you regarding the query and were following up to check if you have found a resolution. If you have identified a solution, we kindly request you to share it with the community, as it may be helpful to others facing a similar issue.

If you find the response helpful, please mark it as the accepted solution and provide kudos, as this will help other members with similar queries.

Thank you.

[v-pnaroju-msft · Accepted solution]

Hi @cstoddard,

Thank you for your inquiry through the Microsoft Fabric Community Forum.

Based on my understanding, the Fabric Runtime is primarily designed for notebook development within Visual Studio Code. This runtime facilitates interactive sessions and immediate feedback, which are integral to notebook workflows. In contrast, Spark Job Definitions (SJDs) are structured for batch processing and are executed on the remote Spark Runtime, aligning with their intended use cases.

Currently, there is no official timeline for integrating the Fabric Runtime directly with SJDs in Visual Studio Code. However, the development team continues to evaluate enhancements to streamline the development experience across different code artifacts.

We encourage you to stay updated on new features and enhancements by regularly checking the Microsoft Fabric Release Plan and the What's New section links provided below:
What's new and planned for Fabric Data Engineering in Microsoft Fabric - Microsoft Fabric | Microsof...
What's New? - Microsoft Fabric | Microsoft Learn

If you find our response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

Thank you.