Forum Discussion
Solved
Notebook Error: CANNOT_OPEN_SOCKET in collect()
Hi Fabric community, I’m running a piece of code in a Fabric notebook that performs a few .collect() and .first() operations on a very small dataset (less than 5 columns). Intermittently, the j...
- Hi,Generally it is not advised to use .collect() even while working with small subset of data as this functions holds the result back to the driver memory. The interaction between the driver and workers while transferring the result makes it more slower as compared to other functions like .take() etc. Its is advisable to use .take(), filter by using .limit() function , cache or persist the results and clear spark cache for releasing driver memoery.Thanks
Hi,
Generally it is not advised to use .collect() even while working with small subset of data as this functions holds the result back to the driver memory. The interaction between the driver and workers while transferring the result makes it more slower as compared to other functions like .take() etc. Its is advisable to use .take(), filter by using .limit() function , cache or persist the results and clear spark cache for releasing driver memoery.
Thanks
Hi navakanth_DE
Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.
Hi navakanth_DE
We wanted to follow up to check if you’ve had an opportunity to review the previous responses. If you require further assistance, please don’t hesitate to let us know.