Forum Discussion
Any integration or tutorials for Spark Connect?
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
>> interactive mode, how do you think it will work in terms of CU calculation. If I run my first command and after it finishes the notebook is not in running state anymo, only the session is active
If the session is active and connected to the cluster then I am 100% certain it would keep accumulating CU's. Ideally the cluster would scale down (via autoscale) to save Microsoft some money. And ideally the dynamically allocated executors woud die off as well to save the customer a bit of money in their notebooks.
... in short, the cluster (custom pool) and VM's are the resources which Microsoft has to keep running at their own expense. It is somewhat fixed. But the CU-meter is accumulated via notebook-compute which is a highly "variable cost". Microsoft probably needs to significantly increase this variable cost that they charge the customer, to ensure that it always covers their own fixed expenses. That is how I understand it.
The notebook will become idle after a period of time and both the cluster and the executors will die. That will stop the billing. And it will stop the expense to Microsoft, in regards to their cluster (custom pool)
Hi dbeavon3,
I did this experiment:
I created a new F64 capacity (so that there is no noise in the capacity metrics app) . I created a new notebook, started a standard session and set the session time out period to be 45 minutes. But I did not run anything on the notebook like you can see below.
In the below metrics app, you can see the consumption which is around 24K CU (s) and the duration to be 2758 seconds which is ~ 46 minutes. So even if we don't run the notebook, there is a CU consumption because the spark session is running (I don't think MS owns this expense as I can see the CU(s) consumed in the Capacity metrics app)
And I agree with your point of running a notebook would be a variable cost ( because there is autoscale and dynamically allocated executors), but there is a fixed cost when running the spark session and based on my understanding, CU(s) is consumed and MS doesn't own it. Unlike Databricks, where you have the option to start the cluster directly, here in Fabric the only way to turn it on is to start a notebook or run a notebook/spark job. In that way, you wouldn't be inadvertently starting a session and accumulating CU(s).
govindarajan_d
>> ... fixed cost when running the spark session and...
When I refer to a "fixed cost". I'm basically referring to the cost of the underlying cluster. We see this on all the other Spark products, but not Fabric.
IE. In the context of HDInsight I may have a cluster that autoscales from 8 to 10 nodes. In such a cluster, my fixed costs are going to be the 8 minimum nodes. No matter what happens, I'm always paying for 8 minimum nodes. (lets say they are ~$10/hr or so, out the door including zookeeper and head nodes). Or about $240 for a dayIn this HDInsight cluster with a fixed cost, I can execute 1000 notebooks in the day that each use 8 executors, or I can execute 1 notebooks for the day that uses 2 executors. Either way I always pay the fixed $10/hr. The costs to my Azure bill will NOT be variable depending on the number of notebooks I may choose to run
But Fabric is extremely different in that Microsoft wants to withold the responsibility for managing cluster; they want to hide that from the customer entirely. They wish to charge us for notebooks instead. I never get to see my cluster cost, because that is not my business anymore. Instead of the cluster itself, Microsoft wants their Fabric customers to focus only on the optimization of their notebooks . Our costs are proportional (variable in a linear way) and they increase based on the number of notebooks we run. Going back to the original example, if we use 1000x8 notebook-hours in Fabric then our CU usage will be about 4000 times more than 1x2 notebook-hours. These CU costs have virtually nothing to do with the underlying cluster anymore. We can see that Microsoft created many layers of abstraction where the customer no longer needs to be aware that VM's are being used, or that there is a physical Spark/Yarn cluster. The only thing they want us to focus on is paying the variable charges that are proportional to the number of notebook-hours. And those variable charges to the customer will be very high, so that it ensures that Microsoft will cover all their own fixed costs that they are incurring behind the scenes.
>> even if we don't run the notebook, there is a CU consumption because the spark session is running
Right ... for the sake of discussing CU costs, you should assume the notebook is "running" whenever the session is connected. It is irrelevant to Microsoft that you don't happen to be evaluating a cell. Eg. you could run a cell for 45 mins that says "time.sleep()" and, assuming the session is connected in either case, it will cost you exactly the same amount as if you were NOT running that sleep cell. Microsoft is charging you for the active executors and drivers in the notebook. I'm guessing you could extend your testing to leverage x1 or x2 or x3 executors and that would increase your costs proportionally for the same duration of 45 mins.