Handling Timeout Issues with GET Requests in Microsoft Fabric

[luis_quiroz]

Hello everyone!

I'm facing an issue while trying to download a zip file from a URL using Microsoft Fabric. My goal is to analyze a CSV file contained within this zip using PySpark. However, I'm encountering a timeout error when making GET requests that take more than 172 seconds (2 minutes and 12 seconds).

The error I receive in my notebook is as follows:

```
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f007b0713c0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /suibase/distribuidorEnergia/creg015_certificados/comer59850_cert3m2024_id85532.sui
MaxRetryError: HTTPConnectionPool(host='www.sui.gov.co', port=80): Max retries exceeded with url: /suibase/distribuidorEnergia/creg015_certificados/comer59850_cert3m2024_id85532.sui (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f007b070670>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
```

This error occurs every 2 minutes and 12 seconds, which coincides with the time it takes for the request.

Below is the Python code I'm using:

```python
url="http://www.sui.gov.co/suibase/distribuidorEnergia/creg015_certificados/comer59850_cert3m2024_id85532..."
import urllib3
from urllib3.util.timeout import Timeout

# Create a PoolManager instance
http = urllib3.PoolManager()

# Specify connection timeout and read timeout separately
connect_timeout = 420 # seconds to establish a connection
read_timeout = 540 # seconds to wait for a response

# Create a Timeout object with separate values
timeout = Timeout(connect=connect_timeout, read=read_timeout)

# Make a request with the specified timeout
response = http.request('GET', url, timeout=timeout)
print(response.status)
print(response.data)
```

I would greatly appreciate any suggestions or solutions you can provide to help resolve this issue and successfully download the zip file. Thank you in advance for your assistance!

[luis_quiroz]

Thank you for your response. Unfortunately, the response isn't working and the ticket created is as follows:

Incident title:	Handling Timeout Issues with GET Requests in Microsoft Fabric (synapse data engineering)
Support request number:	2404290040012156

[v-cboorla-msft]

Hi @luis_quiroz 

The error you provided indicates that the connection is being broken due to the request taking more than 172 seconds to complete. Please try the below approaches which might help you:

Increase Timeout Values: You have already set the connection timeout (connect_timeout) to 420 seconds and read timeout (read_timeout) to 540 seconds. These values should be sufficient for most cases, but since your request is taking longer than expected, you might consider increasing these timeouts further to accommodate for longer download times.

Stream the Response: Instead of waiting for the entire response to be downloaded before processing it, you can stream the response and process it incrementally. This way, you won't hit the timeout limit even if the download takes longer. urllib3 supports streaming responses, which means you can process the data as it's being received without waiting for the entire response.

By streaming the response and processing it incrementally, you should be able to avoid the timeout issue even for longer download times.

I hope this information helps.

If the issue still persists. Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[luis_quiroz]

Thank you for your response. Unfortunately, the response isn't working and the ticket created is as follows:
Incident title: Handling Timeout Issues with GET Requests in Microsoft Fabric (synapse data engineering)

Support request number: 2404290040012156

[v-cboorla-msft]

Hi @luis_quiroz 

Thanks for using Microsoft Fabric Community.

At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.
Appreciate your patience.

Thanks.