Forum Discussion
Any integration or tutorials for Spark Connect?
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
>> interactive mode, how do you think it will work in terms of CU calculation. If I run my first command and after it finishes the notebook is not in running state anymo, only the session is active
If the session is active and connected to the cluster then I am 100% certain it would keep accumulating CU's. Ideally the cluster would scale down (via autoscale) to save Microsoft some money. And ideally the dynamically allocated executors woud die off as well to save the customer a bit of money in their notebooks.
... in short, the cluster (custom pool) and VM's are the resources which Microsoft has to keep running at their own expense. It is somewhat fixed. But the CU-meter is accumulated via notebook-compute which is a highly "variable cost". Microsoft probably needs to significantly increase this variable cost that they charge the customer, to ensure that it always covers their own fixed expenses. That is how I understand it.
The notebook will become idle after a period of time and both the cluster and the executors will die. That will stop the billing. And it will stop the expense to Microsoft, in regards to their cluster (custom pool)
Hi dbeavon3,
Thanks for the explanation. I hope we get more detailed documentation on the Spark CU consumption model!
I will try few more experiments on my side to get a better understanding of what you said!