Forum Discussion
Strange pyspark notebook failure - The Code_Verifier does not match the code_challenge
Hi dbeavon3 ,
AADSTS501481 - Errors starting with AADTS - usually mean something to do with Microsoft Entra ID. Most likely your token which is being used with the user ID, is not working anymore for some reason.
We had a similar issue with D365 Link to Fabric, where an account which was used to create the link had a device removed from its list of devices in Entra ID and out of nowhere we started getting a similar issue which stated the token is not valid. We created a MS ticket and they helped us in solving that.
You can try pausing your capacity and starting it again to see if it helps in resetting the token. If not, I advise you to create a MS ticket.
Still no public -facing help for this error. It keeps surfacing in Fabric.
They seem to be blaming the open source Apache community, but I'm not seeing credible evidence that the bug is someone else's responsibility:
"We checked with the PG team; they provided the update that it is a random issue, and the issue from the gluten/velox side."
I'd guess this bug is something that Microsoft can fix. (Or if they don't want to permanently fix the bug then they can "implement retries" internally.... just as they would ask of their customers all the time).
My RCA (please feel free to disregard, once Microsoft publishes this bug themselves):
I know almost nothing about this bug, aside from the messages that keep popping up. However I'd guess that the bug is not the fault of the apache community project. It is more likely that they are not loading these third-party dependencies successfully, for the sake of the related client code in Fabric. I have seen "warnings" and "errors" in the spark logs that refer to gluten like so ...
ERROR: ld.so: object '/opt/gluten/dep/libjemalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
.. it seems like a pretty serious issue. But wherever I notice these messges in the logs, I belive they are attempting to ignore the error, and plow forward. By trying to suppress a problem like this one, it probably just results in other problems that have even stranger error messages.
It has been almost a whole month since I added my last update. There is no meaningful update from Mindtree or Microsoft. The support is truly bad, and the PG doesn't seem to care how many customers come across this bug.
In theory the PG should be adding it to their "known issues" list, but I think they would rather we all open independent support tickets with Mindtree.
Hi dbeavon3,
I understand the frustration. We too had to wait for almost 2-3 months on fixing a sync delay issue for D365 Link to Fabric. The support team also needs to be trained more on the product to answer questions properly.