Forum Discussion
Any integration or tutorials for Spark Connect?
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
Hi v-prasare
Thanks for the reply.
Assuming I explored the remote connectivity to our Spark environment (pools) via "Spark Connect", how likely would I find a path forward? It sounds like you have very little hope that this would work. Is that right?
It is too bad they would include that announcement about "Spark Connect" in a Fabric offering, if it was not actually available or supported.
I saw that there is a way to download and run python notebooks on the desktop in VS code. Is it possible that this is using "Spark Connect" under the hood? Do you know how that is implemented? Does it support advanced features like UDF declarations in the user's local VS code notebooks?
The main reason why I don't have that much hope for a path forward with "Spark Connect" is because of the way Microsoft is monetizing Spark in Fabric. There is not a tangible cluster that customers can interact with , outside of the context of a notebook. Nor is there any console for monitoring/managing the so-called "pool" that is created behind the scenes of our notebooks. I believe that the only way that the "Spark Connect" would be viable on Fabric is if the Spark cluster had a life of its own. The cluster would need to exist independently of any python notebooks that are running within Fabric . However that would probably conflict with the monetization, and with the way the Spark integration has been introduced into this product. IE. There appears to be no mechanism for Microsoft to send us billing for a Spark cluster, independent from the notebooks that are using the cluster.
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
Hi thanks for the update. I was hoping to play with Spark Connect on Fabric, but I may have to revert to Databricks.
I actually use HDI in Azure more than anything else, but it is stuck on an earlier version of Spark. Hoping that changes in the near future. I love HDI, and wish Microsoft would give it a bit more TLC!
If I had time, I would work on a PoC. But even if it worked, it would probably be fragile, and would not be future-proof. The problem with Fabric Spark is that the cluster is not really a first-class member of the workspace. It is only created behind the scenes for the purpose of a notebook execution ("clients"). All the monetization is handled per notebook. So any of the "server" -oriented features of Spark are probably tucked away under the hood, and there is no surface-area for customers to interact with these features. It is unlikely that "Spark Connect" will be supported, until Microsoft finds a way to bill us for it, and make even more money than what they earn from notebooks. There is a tipping point where I'm guessing they would lose money, if customers are able to manage their own cluster, and successfully optimize the number of notebooks that are able to run on a given day.
Hi dbeavon3,
Spark-connect is good when you are running your own Spark cluster in local network. For example, organizations can run spark cluster at 192.168.1.6 and then an application which wants to use Spark-connect will be present in the same network and can access Spark-connect with local IP address.
But in a propriterary cloud, implementing Spark-connect would be way more difficult to use because of security and data privacy concerns. The monetization is based on CUs used, so it still ties to the Spark Cluster usage rather than notebook.
Hi thanks for the post.
>> ... so it still ties to the Spark Cluster usage rather than notebook.
I don't think this is the case. If cost was tied to the cluster then there would be a cluster monitoring U/I. But in Fabric there is NOT any independent visibility to monitor the cluster, from what I can see. I think the monetization (meter) in Fabric is measured by the notebook. IE. The cost of the CU's will scale up based on the notebook-hours. From what I can tell, it does NOT scale up and down based on vcore-hours (ie. relative to the size of the cluster and time the cluster is running).
The notebook-hours may actually be charged as notebook-compute-hours but the idea is the same. They don't let you cost-optimize at the cluster level, since they want to charge you in relation to your notebooks.
At the end of the day, the spark clusters will cost a lot more than in another simple spark service, like HDInsight or Databricks standard tier.