Forum Discussion

cw88's avatar
cw88
Helper IV
1 year ago
Solved

Notebook: Write data from snowflake (oauth) to lakehouse

I try to get data from snowflake (with oauth authentication) and write it to the default lakehouse with Pyspark:

 

conn = http.client.HTTPSConnection("XXX")
payload = f"XXX"
headers = {'content-type': "application/x-www-form-urlencoded"}
conn.request("POST", "/oauth/token", payload, headers)
res = conn.getresponse()
data = res.read()

json_data = json.loads(data)
TOKEN = json_data["access_token"]
conn = sf.connect(account="XXX",
user=CLIENT_ID + "@clients",
authenticator="oauth",
token=TOKEN,
database="XXX",
schema="XXX",
warehouse="XXX")
cur = conn.cursor()
cur.execute(pSqlquery)

df = cur.fetch_pandas_all()
spark_df = spark.createDataFrame(df)

# Print the DataFrame
spark_df.write.mode("overwrite").format("delta").saveAsTable(pLakehousedestinationtable)

# Close the cursor and connection
cur.close()
conn.close()

When i limit the query to 100 rows everything works fine. But if i want to load > 800 rows, i get the following error:

No CA bundle file is found in the system. Set REQUESTS_CA_BUNDLE to the file.
Failed to fetch the large result set batch 01b8638a-0000-9655-0020-6a8300058006_0%2Fmain%2Fdata_0_0_1 for the 1 th time, backing off for 1s for the reason: '254007: The certificate is revoked or could not be validated: hostname=rhoyejsfcb1stg.blob.core.windows.net'
Traceback (most recent call last):
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/result_batch.py", line 332, in _download
response = session.request("get", **request_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connectionpool.py", line 1058, in _validate_conn
conn.connect()
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/vendored/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/snowflake/connector/ssl_wrap_socket.py", line 93, in ssl_wrap_socket_with_ocsp
raise OperationalError(
snowflake.connector.errors.OperationalError: 254007: The certificate is revoked or could not be validated: hostname=rhoyejsfcb1stg.blob.core.windows.net

Does anyone have an idea what the problem could be?

I use snowflake-connector-python Version 3.12.3 as public library in the environment. 

 

Thanks!

10 Replies