topic Re: HTTPS API Connection Issues in Python Notebooks in Data Engineering

HTTPS API Connection Issues in Python Notebooks

zzthatcher — Tue, 12 Aug 2025 18:08:03 GMT

I get a large amount of data from rest apis.

I use python notebooks to insert it into delta tables.

Several of them recently started having issues where they never get responses. None of the service desks at the data providers have been able to resolve these issues.

The common denominator is that I am requesting the data using python notebooks on fabric.

I think that this is probably a backend routing issue with the servers where the python notebooks are hosted. The frequency of these issues has been increasing to the point where I have to start refactoring code. The issue comes up more often with larger http responses.

I am on US EAST 2.

Luckily I was able to drastically reduce the amount of data per request from one provider without significant refactoring. Fixing the second pipeline where this has come up would require significant and excessive refactoring.

The common symtom is that there have been no changes on my side or the data provider's side, but suddenly the responses are randomly timing out. I already tried resolving it with excessive retries.

Is anyone else running into this?

Re: HTTPS API Connection Issues in Python Notebooks

apturlov — Tue, 12 Aug 2025 19:41:16 GMT

Hi @zzthatcher are you looking for help with your Python code or just generally asking about people experience in similar situation? I don't think a general question like that would give you useful information. Could you clarify what kind of response your looking for and provide more relevant information?

Re: HTTPS API Connection Issues in Python Notebooks

zzthatcher — Tue, 12 Aug 2025 20:39:52 GMT

Hello,

I was hoping to find out if this was a common problem

Looks like I was able to finally resolve one of my issues through downgrading a package.

So this shouldn't actually be common. I would image that my other issue is some low level rate limit. I will remove this post shortly.

Re: HTTPS API Connection Issues in Python Notebooks

apturlov — Wed, 13 Aug 2025 16:16:51 GMT

@zzthatcher well, maybe don't remove this post but instead provide more technical details about the issue and your solution so the others can learn from your experience. Even if the issue is not that common it still happened to you and someone may benefit from your experience.