Fabric Data Engineering plugin for VS Code exceeding pool limits

[TheNoMoreHero]

Hi,

I am trying to setup the Fabric Data Engineering plugin within VS Code. I have been able to successfully load the individual notebooks and such, but I can't get the querying of the datawarehouses to work. See the error below:

[error] Error executing jupyter extension internal startup code def __VSCODE_inject_module():\ndef __VSCODE_call_function(function, callback, data=None): InvalidHttpRequestToLivy: [SparkSettingsMergeValidationError] Settings operation failed due to a validation error: Code = SparkSettingsComputeExceedsPoolLimit, Message = 'The cores or memory you claimed exceeds the limitation of the selected pool, claimed cores: 16, claimed memory: 112, cores limit: 8, memory limit: 56. Please decrease the num of executors or driver/executor size.' . [Root activity id: [redacted] HTTP status code: 400. 
     InvalidHttpRequestToLivy: [SparkSettingsMergeValidationError] Settings operation failed due to a validation error: Code = SparkSettingsComputeExceedsPoolLimit, Message = 'The cores or memory you claimed exceeds the limitation of the selected pool, claimed cores: 16, claimed memory: 112, cores limit: 8, memory limit: 56. Please decrease the num of executors or driver/executor size.' . [Root activity id: [redacted]] HTTP status code: 400.

 As you can see, the request claims more resources than there are available. Even though the environment should be limited to 8 cores and 56 gb of memory. It appears that the request ignores the environment settings.

Is there any way I can adjust these settings so I can get this to work?

[v-csrikanth · Accepted solution]

Hi @TheNoMoreHero 
The error indicates that the Spark job requested more cores and memory than the allowed pool limits.
You need to adjust the Spark configurations to stay within the allowed 8-core, 56GB memory limit.

Modify the Spark Configuration in VS Code:
In your Fabric Data Engineering plugin setup within VS Code, you can manually set the appropriate Spark configurations before running queries.

Try adding the following configurations in your notebook before running queries:
**************************************************************
%%configure -f
{
"executorMemory": "4G",
"executorCores": 2,
"numExecutors": 2,
"driverMemory": "4G",
"driverCores": 2
}
**************************************************************
Restart VS Code after making the changes.
Try running a simple query first to check if the resource request is now within limits.

If the above information helps you, please give us a Kudos and marked the Accept as a solution.

Best Regards,
Community Support Team _ C Srikanth.