topic Re: SSL Verification failed when trying to use OpenAI library in Data Engineering

SSL Verification failed when trying to use OpenAI library

govindarajan_d — Wed, 05 Feb 2025 12:53:23 GMT

I am using an Azure OpenAI model with Python notebook in Fabric.

client = openai.AzureOpenAI( api_key=AZURE_OPENAI_API_KEY, azure_endpoint=AZURE_OPENAI_ENDPOINT, api_version=API_VERSION ) response = client.chat.completions.create()

The above code when run throws SSL issue, although if I try it in local environment it works fine.

The error that appears is:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

I tried the following installations to make sure SSL certificates are:

pip install -U openai
pip install pip-system-certs
pip install --upgrade certifi

Re: SSL Verification failed when trying to use OpenAI library

nilendraFabric — Wed, 05 Feb 2025 13:00:24 GMT

Hello @govindarajan_d

give it a try

import httpx

client = openai.AzureOpenAI(
api_key=AZURE_OPENAI_API_KEY,
azure_endpoint=AZURE_OPENAI_ENDPOINT,
api_version=API_VERSION,
http_client=httpx.Client(verify="/path/to/your/custom_certificate.pem")
)

not recommended for prod, but lets see if this works

Re: SSL Verification failed when trying to use OpenAI library

govindarajan_d — Wed, 05 Feb 2025 18:44:17 GMT

Hi @nilendraFabric,

The code worked, I pointed it to the existing certificate.

client = openai.AzureOpenAI( api_key=AZURE_OPENAI_API_KEY, azure_endpoint=AZURE_OPENAI_ENDPOINT, api_version=API_VERSION, http_client=httpx.Client(verify="/etc/ssl/certs/ca-certificates.crt"), )

But why doesn't it work normally when other libraries like requests work without issues? Also, for production what would you suggest?

Re: SSL Verification failed when trying to use OpenAI library

nilendraFabric — Wed, 05 Feb 2025 19:24:45 GMT

Great it worked @govindarajan_d

I would suggest store pem in Key vault and then use it here.

or store it in a place where it cant be accessed unauthorised.
it often contains sensitive information such as private keys, which must be protected from unauthorized access.

if this is helpful please accept the solution

Re: SSL Verification failed when trying to use OpenAI library

PPuente — Thu, 06 Feb 2025 10:04:22 GMT

Also works with

http_client=httpx.AsyncClient

Re: SSL Verification failed when trying to use OpenAI library

govindarajan_d — Thu, 06 Feb 2025 15:04:06 GMT

@nilendraFabric We are not using custom key for certificate signing. Actually the above certificate is something that comes default with the spark environment (linux). I just pointed the http client to that certificate.

I am just wondering why it doesn't cause issue with requests library when I am hitting HTTPS sites. Strange though!