Forum Discussion
Any integration or tutorials for Spark Connect?
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
Hi thanks for the post.
>> ... so it still ties to the Spark Cluster usage rather than notebook.
I don't think this is the case. If cost was tied to the cluster then there would be a cluster monitoring U/I. But in Fabric there is NOT any independent visibility to monitor the cluster, from what I can see. I think the monetization (meter) in Fabric is measured by the notebook. IE. The cost of the CU's will scale up based on the notebook-hours. From what I can tell, it does NOT scale up and down based on vcore-hours (ie. relative to the size of the cluster and time the cluster is running).
The notebook-hours may actually be charged as notebook-compute-hours but the idea is the same. They don't let you cost-optimize at the cluster level, since they want to charge you in relation to your notebooks.
At the end of the day, the spark clusters will cost a lot more than in another simple spark service, like HDInsight or Databricks standard tier.
Hi dbeavon3,
Well, the reason I said it is cluster because you can have a high-concurrency cluster to which you can attach multiple notebooks and run them parallely. The reason clusters don't have separate monitoring for cost is everything is included in a single billing which Fabric capacity used per second.
You can think of this as a game card that you buy in a mall and you have preloaded points that you can spend on various games. Some games cost you higher than other games and it is upto you how you spend that preloaded points.
In the same way, F2 capacity has 7200 CU-seconds per hour. So, let's say you spin up a standard cluster and assume it costs about 5000 CU seconds every hour. You can attach only one notebook at a time to it. Let's say you spin up a high-concurrency cluster with same size (so same 5000 CU seconds every hour). Now you can attach multiple notebooks to that cluster and run them. But there might be a performance difference depending on the compute requirements of the notebook. If multiple notebooks run with same performanceas standard cluster ( which means standard cluster wasunderutilized) , then there would be a cost-saving. So it would be cluster usage rather than notebook level.