Forum Discussion

dbeavon3's avatar
dbeavon3
Memorable Member
1 year ago
Solved

Any integration or tutorials for Spark Connect?

The docs say that fabric runtime 1.2 supports "Spark Connect", and that there is a python client.  See below.   https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-1-2     .....
data engineering
  • govindarajan_d's avatar
    govindarajan_d
    1 year ago

    Hi dbeavon3 ,

     

    Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system. 

     

    Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.

     

    The following seems true after I read the description from MS site and Spark site. 

    Maybe someone copy/pasted from the OSS docs for Apache Spark.

     

v-prasare's avatar
v-prasare
Community Support
1 year ago

Hi dbeavon3,

Thanks for actively participating in MS Fabric Community Support.

 

Please refer to below video related to spark runtime and latest updates in Fabric: 

Apache Spark Runtimes in Microsoft Fabric – Runtime 1.3 based on Apache Spark 3.5 - YouTube

 

let me know if this helps? looking forward to your feedback!

 

 

Thanks,

Prashanth Are

MS Fabric community support.

 

Did we answer your question? Mark post as a solution, this will help others!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"

dbeavon3's avatar
dbeavon3
Memorable Member
1 year ago

Hi v-prasare 

 

Are you familiar with the "Spark Connect" functionality in Apache Spark?

 

This is a question about a specific feature of a spark environment. 

 

The Fabric announcement that I shared seems to be indicating that this is now a capability of Fabric.  But I have seen no other evidence of that yet.  Have you ever tried to use Spark Connect, within a Fabric environment?

 

Here are the docs, on the OSS/Apache side of things:

https://spark.apache.org/docs/3.4.0/spark-connect-overview.html#spark-connect-overview

 

Any tips would be appreciated.  I checked the video, and it doesn't appear to mention any information that is relevant to "spark connect".