Forum Discussion

navakanth_DE's avatar
navakanth_DE
Frequent Visitor
4 months ago
Solved

Notebook Error: CANNOT_OPEN_SOCKET in collect()

Hi Fabric community,   I’m running a piece of code in a Fabric notebook that performs a few .collect() and .first() operations on a very small dataset (less than 5 columns). Intermittently, the j...
data engineering
  • ati_puri's avatar
    ati_puri
    4 months ago
    Hi,
     
    Generally it is not advised to use .collect() even while working with small subset of data as this functions holds the result back to the driver memory. The interaction between the driver and workers while transferring the result makes it more slower as compared to other functions like .take() etc. Its is advisable to use .take(), filter by using .limit() function , cache or persist the results and clear spark cache for releasing driver memoery.
    Thanks

    navakanth_DE

tayloramy's avatar
tayloramy
Super User
4 months ago

Hi navakanth_DE, 

 

deborshi_nag - I don't think accepting that "this just happens" and advising users to "do less transformations" is an acceptable answer to this problem. 

 

navakanth_DE, I've never encountered this myself, but this is very much an issue if this is happening. I'd recommend opening a support ticket with Microsoft, that way they can dig into the telemetry from your tenant and get to the bottom of exactly what is going on, and if there is a bug in the platform, they can get it on the product team's roadmap to fix. 

 

navakanth_DE's avatar
navakanth_DE
Frequent Visitor
4 months ago

hey tayloramy , I have already raised a MSFT ticket as well long back 4 months ago but even they are not able to provide a proper solution for this .

Only recommendation sugessted by them is to apply a retry mechanism in the code when there is a cocket error .