Forum Discussion
Any integration or tutorials for Spark Connect?
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
We see that in our tenant. "finished" notebooks keep consuming (lots of) CUs until we forcibly close the session.
Hi lbendlin
I'm likely to open the exact same support ticket if this experience happens on our side. I don't suppose you could share your ticket/SR number ( or maybe some tips that I can refer my engineer to find the related ICM ?)
We should probably audit the start and stop times of our notebooks, along with the number of executors used. That should allow us to account for our own notebook-hours, and get refunds when the CU's are calculated improperly. It should be easy in python notebooks to retrieve the start and stop events, and log them to a simple deltatable or something like that.
> until we forcibly close the session.
Yuck. Managing at the session/notebook level is a pain because there can be 100's or 1000's a day. This is the exact reason why customers need a management console for watching our cluster. Assuming the cluster starts and stops when we expect it to, then the notebooks will take care of themselves (they cannot run without a cluster). We shouldn't have to micro-manage them at that level of granularity. Yet I don't really trust Microsoft to micro-manage the notebook-hours as closely as required. They don't care if a session is closed; and they will typically point the finger back at their customer and blame the problem on some python library we used, or some "time.sleep()" statement or whatever.