HTTPS API Connection Issues in Python Notebooks

[zzthatcher]

I get a large amount of data from rest apis.

I use python notebooks to insert it into delta tables. 

Several of them recently started having issues where they never get responses. None of the service desks at the data providers have been able to resolve these issues.

The common denominator is that I am requesting the data using python notebooks on fabric. 

I think that this is probably a backend routing issue with the servers where the python notebooks are hosted. The frequency of these issues has been increasing to the point where I have to start refactoring code. The issue comes up more often with larger http responses. 

I am on US EAST 2. 

Luckily I was able to drastically reduce the amount of data per request from one provider without significant refactoring. Fixing the second pipeline where this has come up would require significant and excessive refactoring. 

The common symtom is that there have been no changes on my side or the data provider's side, but suddenly the responses are randomly timing out. I already tried resolving it with excessive retries.

Is anyone else running into this?

[apturlov]

Hi @zzthatcher are you looking for help with your Python code or just generally asking about people experience in similar situation? I don't think a general question like that would give you useful information. Could you clarify what kind of response your looking for and provide more relevant information?

[zzthatcher · Accepted solution]

Hello,

I was hoping to find out if this was a common problem

Looks like I was able to finally resolve one of my issues through downgrading a package. 

So this shouldn't actually be common. I would image that my other issue is some low level rate limit. I will remove this post shortly.

[apturlov]

@zzthatcher well, maybe don't remove this post but instead provide more technical details about the issue and your solution so the others can learn from your experience. Even if the issue is not that common it still happened to you and someone may benefit from your experience.