Native execution engine enabled by default in fabric spark?

[dbeavon3]

I keep running into tons of spark bugs, like the one described here and more:

https://learn.microsoft.com/en-us/fabric/known-issues/known-issue-1028-slow-spark-session-startup-na...

The error message we encounter often looks like so:

The Code_Verifier does not match the code_challenge supplied in the authorization request

The docs say this feature is in preview:

Native execution engine for Fabric Spark - Microsoft Fabric | Microsoft Learn

Is this stuff enabled for everyone else? Or is my tenant being treated as a guinea pig?

Given that this is in preview and there are still bugs, why is it taking effect in my environment?

[v-pagayam-msft]

Hi @dbeavon3 ,
We wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.If our answer resolved your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.If you need any further assistance, feel free to reach out.

Thank you for being a valued member of the Microsoft Fabric Community Forum!

[v-pagayam-msft]

Hi @dbeavon3 

I wanted to check in on your situation regarding the issue. Have you resolved it? If you have, please consider marking the reply that helped you as Accept as solution or sharing your solution. It would be greatly appreciated by others in the community who may have the same question.

Thank you for being a valued member of the Microsoft Fabric Community Forum!

[v-pagayam-msft]

Hi @dbeavon3  ,
Thank you for reaching out to us on the Microsoft Fabric Community Forum.

The term "public preview" means the feature is available for all users to test, but it may still be under development and subject to changes. The announcement of availability at no additional cost likely refers to the feature being accessible during this preview phase without extra charges.

Given that the feature is in public preview, it's possible that your environment has been included to gather diverse usage data and feedback. If the Native Execution Engine is causing issues in your environment, you have the option to disable it at both the environment and notebook/job levels.

• Environment Level: Navigate to your environment settings in the Microsoft Fabric portal. Under the "Acceleration" tab, you may find a toggle to enable or disable the Native Execution Engine.
   
• Notebook or Job Level: Within your notebooks or Spark job definitions, you can control this feature by setting the Spark configuration spark.native.enabled to false at the beginning of your script.

  Please refer below:

  manage-compute

  native-execution-engine-overview

  Native Execution Engine available at no additional cost!

  If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details, always happy to help.
  Thank you.
  
  Regards,
  Pallavi.

[dbeavon3]

Hi @v-pagayam-msft 

Thanks for the info. Is this stuff enabled for everyone else? (Can you check on your side?)

The purpose is to find out if my tenant being treated as a guinea pig.

My purpose is not to serve Microsoft in the capacity of a beta-tester.  If Microsoft wishes to "gather diverse usage data and feedback", then that should be done in an opt-in basis, not opt-out.

NOTE: If something is in preview, then it is probably even less reliable than normal, and even less adequately supported by Mindtree.  These are the things that I would NOT want to be enabled by default in our environment.

Please let me know about your environment as well.

[v-pagayam-msft]

Hi @dbeavon3 ,
Thank you for the response.

The Native Execution Engine for Fabric Spark is in public preview and has been made widely available, rather than being selectively enabled for specific tenants. While it is not enabled in my environment, it is likely active for many users. If you prefer to disable it, you may do so at the environment level (Fabric portal → Acceleration tab) or at the notebook/job level, as mentioned earlier. Refer the link for more information

Hope this helps.If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you for being a valued member in Microsoft Fabric Community Forum!

[dbeavon3]

Hi @v-pagayam-msft 

Thanks for the clarification.

As a customer it is pretty frustrating to be automatically enrolled as a beta tester.  The support tickets with Mindtree take weeks, since the PG is totally uninterested in hearing from customers about these Spark bugs in their own "public preview" features.  

Next time I'm in Fabric, I will try to remember to find and disable preview features which are enabled by default.  Thankfully it one is not the only spark environment where I deploy our solutions.

[v-pagayam-msft]

Hi @dbeavon3 ,

I apologize for the inconvenience caused. I understand the feature that was enabled by default. Microsoft is continuously working to enhance the Native Execution Engine and address reported issues to improve stability and performance.

If this post helps, please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thank you for being a part of the Microsoft Fabric Community Forum!

[dbeavon3 · Accepted solution]

The answer is that Microsoft should not be enabling preview features by default.  Fabric is already buggy enough without having to deal with unexpected errors from the preview features as well.

Customers don't want to spend additional time working with Mindtree support, as a results of untested features that are being secretly enabled in our workspaces.

Here is one of the known issues that refers to this error message:

[v-pagayam-msft]

Hi @dbeavon3 ,

Thank you for your response. I understand your concerns, and I kindly request your patience while the issue is being addressed. The Microsoft team is actively working on improvements.

Thank you for your cooperation. Have a great day.