topic Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge in Data Engineering

Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

dbeavon3 — Mon, 06 Jan 2025 21:51:34 GMT

Can someone please provide an English translation of the following message in a Fabric Spark UI.

Description: AADSTS501481: The Code_Verifier does not match the code_challenge supplied in the authorization request.

The "spark monitor" screen in fabric has very little info. It basically just says "something bad happened" on an executor.

I proceeded into the spark UI, and got this other error when trying to reach executor logs in that portal:

It gave me this message when clicking on stderr/stdout links. I really don't think this error has anything to do with the reason the original spark notebook died. It is an issue with the Spark UI itself; and it is preventing us from investigating the original error.

Users who don't know better may think this message is actually coming out of the stderr/stdout logs; but that is not likely. Instead, the message is telling us that Fabric can't open the stderr/stdout logs.

Please avoid the temptation of copy/pasting a response from chat GPT/copilot/whatever. Those answers are really poor, where Fabric is concerned. If I wanted that content, then I would go use those resources myself.

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

govindarajan_d — Thu, 09 Jan 2025 03:27:47 GMT

Hi @dbeavon3 ,

AADSTS501481 - Errors starting with AADTS - usually mean something to do with Microsoft Entra ID. Most likely your token which is being used with the user ID, is not working anymore for some reason.

We had a similar issue with D365 Link to Fabric, where an account which was used to create the link had a device removed from its list of devices in Entra ID and out of nowhere we started getting a similar issue which stated the token is not valid. We created a MS ticket and they helped us in solving that.

You can try pausing your capacity and starting it again to see if it helps in resetting the token. If not, I advise you to create a MS ticket.

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

dbeavon3 — Thu, 09 Jan 2025 01:21:06 GMT

Thanks @govindarajan_d

Thanks, The problem appears to be in the spark UI, and it eventually went away.

There were other strange issues in our capacity that day. I'm guessing there may have been a region-specific outage or degradation. We are in the North Central US region, and I think Microsoft doesn't have as high of a regard for certain regions like this one. I wouldn't expect a public communication about service degradation in the North Central region.

I will keep an eye on it. I have a ticket open with Mindtree, but I'm guessing they won't be able to give me more information than what you just did.

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

govindarajan_d — Thu, 09 Jan 2025 03:25:52 GMT

@dbeavon3 I hope MS provides a RCA. We have been waiting on our own RCA for past 3 weeks! 👀

From different communications I had with MS, I understood there is a list of lower regions, but exactly what makes up that list, I am yet to find out!

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

dbeavon3 — Fri, 17 Jan 2025 14:29:46 GMT

Update... I was (finally) able to get the PG to accept an ICM about this error. This PG does not want to receive word of their bugs. I'm guessing it is because they already know about them, or because they already have more bugs than they can handle.

Unfortunately I do not have a "unified" contract and this bug is not likely to be fixed very soon. Would be grateful if anyone with a unified contract encounters this and wants to hasten a fix (or perhaps just add the issue to the "known issues" list). The ICM is associated with this SR, and any FTE at Microsoft in the support organization should be able to find and verify the ICM:
"SR.2501060040015225.MGR.S.D."

The time commitment involved in raising this topic with Microsoft should be minimal for another customer, given the ICM is already on their list (towards the bottom). It was with Mindtree for about a week, and we have already done the majority of the work. The only part remaining is to encourage a fix from Microsoft.

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

dbeavon3 — Wed, 22 Jan 2025 18:57:48 GMT

Still no public -facing help for this error. It keeps surfacing in Fabric.

They seem to be blaming the open source Apache community, but I'm not seeing credible evidence that the bug is someone else's responsibility:

"We checked with the PG team; they provided the update that it is a random issue, and the issue from the gluten/velox side."

I'd guess this bug is something that Microsoft can fix. (Or if they don't want to permanently fix the bug then they can "implement retries" internally.... just as they would ask of their customers all the time).

My RCA (please feel free to disregard, once Microsoft publishes this bug themselves):

I know almost nothing about this bug, aside from the messages that keep popping up. However I'd guess that the bug is not the fault of the apache community project. It is more likely that they are not loading these third-party dependencies successfully, for the sake of the related client code in Fabric. I have seen "warnings" and "errors" in the spark logs that refer to gluten like so ...

ERROR: ld.so: object '/opt/gluten/dep/libjemalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

.. it seems like a pretty serious issue. But wherever I notice these messges in the logs, I belive they are attempting to ignore the error, and plow forward. By trying to suppress a problem like this one, it probably just results in other problems that have even stranger error messages.

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

dbeavon3 — Tue, 18 Feb 2025 01:50:48 GMT

It has been almost a whole month since I added my last update. There is no meaningful update from Mindtree or Microsoft. The support is truly bad, and the PG doesn't seem to care how many customers come across this bug.

In theory the PG should be adding it to their "known issues" list, but I think they would rather we all open independent support tickets with Mindtree.

Re: Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge

govindarajan_d — Thu, 20 Feb 2025 03:45:17 GMT

Hi @dbeavon3,

I understand the frustration. We too had to wait for almost 2-3 months on fixing a sync delay issue for D365 Link to Fabric. The support team also needs to be trained more on the product to answer questions properly.