Forum Discussion

dbeavon3's avatar
dbeavon3
Memorable Member
1 year ago
Solved

Any integration or tutorials for Spark Connect?

The docs say that fabric runtime 1.2 supports "Spark Connect", and that there is a python client.  See below.   https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-1-2     .....
data engineering
  • govindarajan_d's avatar
    govindarajan_d
    1 year ago

    Hi dbeavon3 ,

     

    Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system. 

     

    Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.

     

    The following seems true after I read the description from MS site and Spark site. 

    Maybe someone copy/pasted from the OSS docs for Apache Spark.

     

v-prasare's avatar
v-prasare
Community Support
1 year ago

dbeavon3, Sorry for delayed response and thanks for your patience. I realized that I misunderstood your original question about Spark Connect and shared a video that focused more on Spark Runtime updates rather than addressing the specific feature you were inquiring about. I apologize for any confusion this may have caused.

 

 You're absolutely right that Spark Connect is a significant feature introduced in Apache Spark 3.4, allowing remote clients to interact with Spark clusters. However, the available Microsoft Fabric documentation and release notes (including the Runtime 1.3 update), there hasn’t been any explicit mention of Spark Connect being officially supported in Fabric yet.

 The video and docs focus primarily on performance optimizations and runtime upgrades, but Spark Connect as a feature seems to be more of an upstream Apache Spark development at this stage, without clear Fabric implementation details. I’ll keep a close eye on any future announcements or documentation updates from Microsoft that might confirm Spark Connect support. Please write your feedback and ideas on fabric here.

In the meantime, exploring alternative connectors and APIs that are currently available within Fabric such as the SQL Analytics endpoint for remote access to Spark datasets.

 

 

 

Thanks,

Prashanth Are

MS Fabric community support.

 

Did we answer your question? Mark post as a solution, this will help others!

your feedback is valuable to us, don't forget to drop me a "Kudos"