Forum Discussion

navakanth_DE's avatar
navakanth_DE
Frequent Visitor
4 months ago
Solved

Notebook Error: CANNOT_OPEN_SOCKET in collect()

Hi Fabric community,   I’m running a piece of code in a Fabric notebook that performs a few .collect() and .first() operations on a very small dataset (less than 5 columns). Intermittently, the j...
data engineering
  • ati_puri's avatar
    ati_puri
    4 months ago
    Hi,
     
    Generally it is not advised to use .collect() even while working with small subset of data as this functions holds the result back to the driver memory. The interaction between the driver and workers while transferring the result makes it more slower as compared to other functions like .take() etc. Its is advisable to use .take(), filter by using .limit() function , cache or persist the results and clear spark cache for releasing driver memoery.
    Thanks

    navakanth_DE