msgraph in notebook fails with new environment [SSL: CERTIFICATE_VERIFY_FAILED]

[renaudjmathieu]

Hi, i use msgraph in notebooks in Fabric.

I have a Fabric environment : 1.2 (Spark 3.4, Delta 2.4) with libraries azure-identity and msgraph

everything works fine.

When I duplicate this environement (the reason is I need to move the environment to another workspace) and change my notebook to use this new seemingly identical environment, i now get the following error : 

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

I tried the following, but to no success : 
https://community.fabric.microsoft.com/t5/Fabric-platform/Using-msggraph-sdk-in-Fabric-notebooks/m-p...

Any idea why it's happening when my notebooks are connected to one Fabric environment but not when connected to the other one that's identical?
Also, any ideas on how to solve it? Do I need to authorize the new environment or something? I don't remember doing it for the old environment but not 100% sure. 

Any help would be appreciated! Thanks!

[renaudjmathieu · Accepted solution]

Ok so the solution for me was : 

• New environment with msgraph-sdk==1.22.0 but not with azure-identity (this caused a dotenv issue).
• Add a cell (because this need to run each time the session is restarted or else I get the error [SSL: CERTIFICATE_VERIFY_FAILED]) at the beginning of my notebook, before initialize the graph client, containing the following : 
  1. os.environ["SSL_CERT_FILE"] = "/home/trusted-service-user/cluster-env/clonedenv/ssl/cert.pem"
  2. os.environ["SSL_CERT_DIR"] = "/home/trusted-service-user/cluster-env/clonedenv/ssl/certs"

This seems to do the trick!

Thanks again everyone for the help. 

[v-hashadapu]

Hi @renaudjmathieu , Thank you for reaching out to the Microsoft Community Forum.

Please let us know if your issue is solved. If it is, consider marking the answer 'Accept as Solution', so others with similar queries can find it easily. If not, please share the details.
Thank you.

[v-hashadapu]

Hi @renaudjmathieu, thank you for reaching out to the Microsoft Fabric Community Forum.

Please check & try the following:

1. Even though ssl.get_default_verify_paths() returns the same paths in both environments, the actual certificate files may differ. Manually verify the certificate file exists and is valid and inspect its contents. If the file is missing or corrupt, you may need to regenerate or reinstall the certificates.
2. If the new environment is missing some required certificates, updating them might help.
3. If the new environment is in a different workspace, it might be subject to different network policies. Try accessing the Microsoft Graph API directly. If this fails, check firewall settings or network policies in the new workspace.
4. Try explicitly pointing Python to the certificate file. Restart the session after setting this.
5. Try a clean Fabric environment and install dependencies from scratch.
6. Compare pip list output between the old and new environments to check for version mismatches.
7. Check if authentication needs re-authorization, especially if the environment is in a different workspace.

We sincerely appreciate your continued support within the community and thank you for your dedicated efforts in resolving issues @nilendraFabric.

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.

[renaudjmathieu]

Hi, thanks for the answer!
the problem is not solve but I may have some clues now. I think its related to "python-dotenv"

Both environment are in the same workspace.
I tried and retried creating a new environment and I always get the same error.

The new environments I created use the same Public Libraries as the old one, and I compared Built-in Libraries and they are the exact same.

I used "!pip list" to compare packages and it looks like new envionments are created with "python-dotenv" whereas the old, working environment didn't include it. 

It's probably related to this? Still looking into it..

[v-hashadapu]

Hi @renaudjmathieu , thank you for reaching out to the Microsoft Fabric Community Forum.

Sorry to know it didn’t help you. If the new environment includes python-dotenv while the old one does not, it's likely that python-dotenv is interfering with the SSL configuration or environment variable loading. 

Please check the following:

1. Run the following in your notebook to see if python-dotenv is modifying environment variables:

import dotenv

print(dotenv.find_dotenv())  # Check if a .env file is being loaded

print(dotenv.load_dotenv())  # See if it's modifying environment variables

*If a .env file is present, inspect its contents to ensure it’s not altering SSL-related settings.

2. Since your old environment didn’t have python-dotenv, removing it might resolve the issue, then restart the session and test again:

!pip uninstall -y python-dotenv

3. If uninstalling python-dotenv fixes the issue, but you still experience SSL problems, explicitly set the SSL certificate paths in your notebook:

import os

os.environ["SSL_CERT_FILE"] = "/home/trusted-service-user/cluster-env/clonedenv/ssl/cert.pem"

os.environ["SSL_CERT_DIR"] = "/home/trusted-service-user/cluster-env/clonedenv/ssl/certs"

4. If uninstalling python-dotenv solves the issue, but you want a more permanent fix, create a new Fabric environment without it. This ensures future notebooks don’t encounter the same issue.

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.

[renaudjmathieu]

Hi! Thanks again for pointing me in the right direction!
It seems dotenv gets install automatically when I include more recent versions of the msgraph-core/msgraph-sdk librairies

I'm still investigating the exact cause of this and will also try to explicitly set the SSL certificate paths like you suggested, if need be.
I will update this ticket once I'm done.

Thanks again! Have a good day

[v-hashadapu]

Hi , Thank you for reaching out to the Microsoft Community Forum.

Happy to help. Thanks for the update. Please share the insights here once it is solved, which will help others with similar queries.

Have a great day. Hope your issue’s solved soon.

Also, please consider marking the answer 'Accept as Solution' so others with similar queries may find it more easily. 
Thank you.

[renaudjmathieu]

Hi, thanks for the help.

in both cases, the old working environment and the new one, the ssl.get_default_verify_paths() is the same : 
DefaultVerifyPaths(cafile='/home/trusted-service-user/cluster-env/clonedenv/ssl/cert.pem', capath=None,
openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/home/trusted-service-user/cluster-env/clonedenv/ssl/cert.pem',
openssl_capath_env='SSL_CERT_DIR', openssl_capath='/home/trusted-service-user/cluster-env/clonedenv/ssl/certs')

sadly, setting the environment variables as described above, and restarting the session, does not seem to help. I get the same error.

[nilendraFabric]

Try these

Check Python’s default certificate path

import ssl
print(ssl.get_default_verify_paths())

Set Environment Variables (as suggested in community discussions):

import os
os.environ["SSL_CERT_DIR"] = "/etc/pki/ca-trust/extracted/openssl:/opt/olcclient"