Forum Discussion
Microsf Fabric - How to handle timeout when making GET requests in Microsoft Fabric?
¡Hola everyone!
I'm facing an issue while trying to download a zip file from a URL using Microsoft Fabric. My goal is to parse a CSV file contained in this zip using PySpark. However, I encounter a timeout error when making GET requests that take longer than 172 seconds (2 minutes and 12 seconds).
The error I'm getting in my notebook is as follows:
```
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f007b0713c0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /suibase/distribuidorEnergia/creg015_certificados/comer59850_cert3m2024_id85532.sui
MaxRetryError: HTTPConnectionPool(host='www.sui.gov.co', port=80): Max retries exceeded with url: /suibase/distribuidorEnergia/creg015_certificados/comer59850_cert3m2024_id85532.sui (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f007b070670>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
```
This error occurs every 2 minutes and 12 seconds, which coincides with the time the request is taking.
I'm attaching the Python code I'm using:
```python
url="http://www.sui.gov.co/suibase/distribuidorEnergia/creg015_certificados/comer59850_cert3m2024_id85532.sui"
import urllib3
from urllib3.util.timeout import Timeout
# Create a PoolManager instance
http = urllib3. PoolManager()
# Specify the timeout for connection and reading separately
connect_timeout = 420# seconds to establish a connection
read_timeout = 540# seconds to wait for a response
# Create a Timeout Object with Separate Values
timeout = Timeout(connect=connect_timeout, read=read_timeout)
# Make a request with the specified timeout
response = http.request('GET', url, timeout=timeout)
print(response.status)
print(response.data)
'''
I would very much appreciate any suggestions or solutions you can give me to solve this problem and be able to properly download the zip file. Thank you in advance for your help!
3 Replies
- AnonymousNot applicable
1.Based on your description, you can consider to extend your setting of the connect_timeout and read_timeout to test if they can receive the data.
2.Check that if your dataset is large that it will affect the performance, you can consider to optimize the data model.
Best Regards!
Yolo Zhu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
- Syndicate_Admin
Administrator
Hello
I appreciate your prompt response. I'd like to confirm that I've already configured the connect_timeout and read_timeout parameters in my GET request, setting connect_timeout to 420 seconds and read_timeout to 540 seconds. Despite this, the problem persists, as the request keeps stopping when it reaches 2 minutes and 12 seconds.
The size of the zip file I'm trying to download is only 23KB, so it shouldn't significantly affect performance. However, it seems that something internal is disrupting the process.
I'm wondering if there's a way to configure the notebook network in Microsoft Fabric to fix this problem.
I appreciate in advance any additional help you can give me!
- Syndicate_Admin
Administrator
I tried to fix this by adjusting spark.network.timeout, but the process still stops at 2 minutes and 12 seconds. I can't find where this time limit is set. Do you have any idea where it might be defined?