Forum Discussion
Any integration or tutorials for Spark Connect?
Hi dbeavon3 ,
Based on my understanding, since Spark connect requires remote connectivity, it needs a hostname which would be the IP address of the Spark Context. And since there is no authentication mechanism invovled with Spark-connect unless you manually setup a re-direction URL mechanism (authentication proxy), I don't believe Fabric will allow that level of configuration in their cloud system.
Using Managed Virtual networks with Fabric, you might get the URL of the Spark context and use it, but again this is just my assumption and as you said, there is no documentation, it is difficult to validate unless we do a PoC.
The following seems true after I read the description from MS site and Spark site.
Maybe someone copy/pasted from the OSS docs for Apache Spark.
You said:
>> So, let's say you spin up a standard cluster and assume it costs about 5000 CU seconds every hour.
I think there are some bad assumptions here. Do you have any supporting links to say that a given-sized "cluster" has a fixed CU cost per hour? I have not found that because I don't think it exists in Fabric. (It would be true in all the other Spark platforms but it is not true in Fabric.)
I suspect the information you have is wrong, and/or it is subject to change. It is probably something that was told to you verbally by a Microsoft salesperson...
First of all the word "cluster" in Fabric is replaced with "pool" which is presented to users as metadata, rather than a physical entity. The subtle change in terminology will create ambiguity and that works in Microsoft's favor. Secondly there is no management console for a "cluster" in Fabric. If CU costs could truly be optimized in the way that you described then having a management console would be a very high priority. There is such a console in the Synapse and Databricks and HDI platforms. But in Fabric there is NOT likely to be one soon, since it is NOT directly relevant to Fabric cost-management, and since Microsoft wants their platform to be "easy", and don't want users to concern themselves with these superfluous implementation details.
Thirdly, the CU meters for Spark notebooks are NEVER presented in the terms that you described ("CU seconds per hour per cluster"). As I said, the accounting is decrementing CU's is based on notebook-hours, or notebook-compute-hours. You can visualize this in their "Capacity Metrics App" (... such as it is, given that it seems to be the only administration tool available for Spark in Fabric). See below.
We can probably agree that optimization is accomplished by reducing the length of time that notebooks run, and using fewer/smaller executors for the synapse notebooks. Doing these things will allow us to see the positive changes in the "capacity metrics app".
Where we don't agree is when you say that there is a fixed cost at the "cluster" level, and that customers can optimize our workloads at that level. There is no way for us to get operating leverage, by way of a fixed cluster cost; because Fabric billing does not happen that way. The billing happens exclusively via the variable costs per notebook-hour.
Hi dbeavon3,
I can be wrong on this since there is no definite documentation on how actually the CUs are calculated for Spark! This is my own logical interpretation based on my experience with Spark.
The logic of Spark session time or notebook/job running time is same in case of non-interactive mode as session is terminated soon after finishing the task.
But in interactive mode, how do you think it will work in terms of CU calculation. If I run my first command and after it finishes the notebook is not in running state anymo, only the session is active. So, in that case is there any CU consumption? If no, then it means MS is running a compute without CU consumption. (Coming from databricks environment, as soon as you switch on a compute, you are charged no matter whether you attach a notebook or not 🙂 ) I am just wondering how would MS run a compute without CU consumption, because the session needs to be active for the notebook to maintain state.
A Microsoft person can have a better idea about this and due to lack of proper documentation, speculation is what I can do at this moment!