Forum Discussion

navakanth_DE's avatar
navakanth_DE
Frequent Visitor
4 months ago
Solved

Notebook Error: CANNOT_OPEN_SOCKET in collect()

Hi Fabric community,   I’m running a piece of code in a Fabric notebook that performs a few .collect() and .first() operations on a very small dataset (less than 5 columns). Intermittently, the j...
data engineering
  • ati_puri's avatar
    ati_puri
    4 months ago
    Hi,
     
    Generally it is not advised to use .collect() even while working with small subset of data as this functions holds the result back to the driver memory. The interaction between the driver and workers while transferring the result makes it more slower as compared to other functions like .take() etc. Its is advisable to use .take(), filter by using .limit() function , cache or persist the results and clear spark cache for releasing driver memoery.
    Thanks

    navakanth_DE

tayloramy's avatar
tayloramy
Super User
4 months ago

Hi navakanth_DE, 

 

I have not encountered this in my fabric environments before. Can you tell us what your data sources are? 

 

Is this an issue when reading the data from source, when writing it to a fabric datastore as a target, or is this an issue in processing the data once it's already been loaded to a spark dataframe? 

 

navakanth_DE's avatar
navakanth_DE
Frequent Visitor
4 months ago

Hey tayloramy ,

Getting this error while i am processing the data from a dataframe . For example , read the data from delta lake and trying to get a record using a collect/first function from the dataframe