Forum Discussion
Any integration or tutorials for Spark Connect?
The docs say that fabric runtime 1.2 supports "Spark Connect", and that there is a python client. See below.
https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-1-2
... see section:
Is this a documentation bug? Is there an actual integration with "Spark Connect"? Nothing comes up in my google results so I'm not getting my hopes up yet.
We have some apps that would benefit from using using a spark server. It would be great to initiate our Spark workloads to run on Fabric, by way of our on-premise applications and services. Most of our on-premise applications are .Net -based. But we might be able to use python as a bridge to initate the Spark workloads, if that type of client is available.
I would love to see a tutorial or example, but haven't found any yet. The docs above may not actually apply to Fabric. Maybe someone copy/pasted from the OSS docs for Apache Spark.
If this was available, it seems like it would be incompatible with the approach that Microsoft is using for the billing of our Spark workloads in Fabric. The billing seems to be tied to client notebooks hosted within the Fabric environment. But Spark Connect seems like a client/server technology, and the client notebooks might be running remotely.
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
17 Replies
dbeavon3, Sorry for delayed response and thanks for your patience. I realized that I misunderstood your original question about Spark Connect and shared a video that focused more on Spark Runtime updates rather than addressing the specific feature you were inquiring about. I apologize for any confusion this may have caused.
You're absolutely right that Spark Connect is a significant feature introduced in Apache Spark 3.4, allowing remote clients to interact with Spark clusters. However, the available Microsoft Fabric documentation and release notes (including the Runtime 1.3 update), there hasn’t been any explicit mention of Spark Connect being officially supported in Fabric yet.
The video and docs focus primarily on performance optimizations and runtime upgrades, but Spark Connect as a feature seems to be more of an upstream Apache Spark development at this stage, without clear Fabric implementation details. I’ll keep a close eye on any future announcements or documentation updates from Microsoft that might confirm Spark Connect support. Please write your feedback and ideas on fabric here.
In the meantime, exploring alternative connectors and APIs that are currently available within Fabric such as the SQL Analytics endpoint for remote access to Spark datasets.
Thanks,
Prashanth Are
MS Fabric community support.
Did we answer your question? Mark post as a solution, this will help others!
your feedback is valuable to us, don't forget to drop me a "Kudos"
Hi v-prasare
Thanks for the reply.Assuming I explored the remote connectivity to our Spark environment (pools) via "Spark Connect", how likely would I find a path forward? It sounds like you have very little hope that this would work. Is that right?
It is too bad they would include that announcement about "Spark Connect" in a Fabric offering, if it was not actually available or supported.
I saw that there is a way to download and run python notebooks on the desktop in VS code. Is it possible that this is using "Spark Connect" under the hood? Do you know how that is implemented? Does it support advanced features like UDF declarations in the user's local VS code notebooks?
The main reason why I don't have that much hope for a path forward with "Spark Connect" is because of the way Microsoft is monetizing Spark in Fabric. There is not a tangible cluster that customers can interact with , outside of the context of a notebook. Nor is there any console for monitoring/managing the so-called "pool" that is created behind the scenes of our notebooks. I believe that the only way that the "Spark Connect" would be viable on Fabric is if the Spark cluster had a life of its own. The cluster would need to exist independently of any python notebooks that are running within Fabric . However that would probably conflict with the monetization, and with the way the Spark integration has been introduced into this product. IE. There appears to be no mechanism for Microsoft to send us billing for a Spark cluster, independent from the notebooks that are using the cluster.
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
Hi dbeavon3,
Thanks for actively participating in MS Fabric Community Support.
Please refer to below video related to spark runtime and latest updates in Fabric:
Apache Spark Runtimes in Microsoft Fabric – Runtime 1.3 based on Apache Spark 3.5 - YouTube
let me know if this helps? looking forward to your feedback!
Thanks,
Prashanth Are
MS Fabric community support.
Did we answer your question? Mark post as a solution, this will help others!
If our response(s) assisted you in any way, don't forget to drop me a "Kudos"
Hi v-prasare
Are you familiar with the "Spark Connect" functionality in Apache Spark?
This is a question about a specific feature of a spark environment.
The Fabric announcement that I shared seems to be indicating that this is now a capability of Fabric. But I have seen no other evidence of that yet. Have you ever tried to use Spark Connect, within a Fabric environment?
Here are the docs, on the OSS/Apache side of things:
https://spark.apache.org/docs/3.4.0/spark-connect-overview.html#spark-connect-overview
Any tips would be appreciated. I checked the video, and it doesn't appear to mention any information that is relevant to "spark connect".
