Forum Discussion
Native execution engine enabled by default in fabric spark?
The answer is that Microsoft should not be enabling preview features by default. Fabric is already buggy enough without having to deal with unexpected errors from the preview features as well.
Customers don't want to spend additional time working with Mindtree support, as a results of untested features that are being secretly enabled in our workspaces.
Here is one of the known issues that refers to this error message:
Hi dbeavon3 ,
Thank you for reaching out to us on the Microsoft Fabric Community Forum.
The term "public preview" means the feature is available for all users to test, but it may still be under development and subject to changes. The announcement of availability at no additional cost likely refers to the feature being accessible during this preview phase without extra charges.
Given that the feature is in public preview, it's possible that your environment has been included to gather diverse usage data and feedback. If the Native Execution Engine is causing issues in your environment, you have the option to disable it at both the environment and notebook/job levels.
- Environment Level: Navigate to your environment settings in the Microsoft Fabric portal. Under the "Acceleration" tab, you may find a toggle to enable or disable the Native Execution Engine.
- Notebook or Job Level: Within your notebooks or Spark job definitions, you can control this feature by setting the Spark configuration spark.native.enabled to false at the beginning of your script.
Please refer below:
native-execution-engine-overview
Native Execution Engine available at no additional cost!
If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Regards,
Pallavi.
Hi Anonymous
Thanks for the info. Is this stuff enabled for everyone else? (Can you check on your side?)The purpose is to find out if my tenant being treated as a guinea pig.
My purpose is not to serve Microsoft in the capacity of a beta-tester. If Microsoft wishes to "gather diverse usage data and feedback", then that should be done in an opt-in basis, not opt-out.
NOTE: If something is in preview, then it is probably even less reliable than normal, and even less adequately supported by Mindtree. These are the things that I would NOT want to be enabled by default in our environment.
Please let me know about your environment as well.
Hi dbeavon3 ,
Thank you for the response.
The Native Execution Engine for Fabric Spark is in public preview and has been made widely available, rather than being selectively enabled for specific tenants. While it is not enabled in my environment, it is likely active for many users. If you prefer to disable it, you may do so at the environment level (Fabric portal → Acceleration tab) or at the notebook/job level, as mentioned earlier. Refer the link for more information
Hope this helps.If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you for being a valued member in Microsoft Fabric Community Forum!Hi Anonymous
Thanks for the clarification.
As a customer it is pretty frustrating to be automatically enrolled as a beta tester. The support tickets with Mindtree take weeks, since the PG is totally uninterested in hearing from customers about these Spark bugs in their own "public preview" features.
Next time I'm in Fabric, I will try to remember to find and disable preview features which are enabled by default. Thankfully it one is not the only spark environment where I deploy our solutions.