Forum Discussion
Notebook Error: CANNOT_OPEN_SOCKET in collect()
Hi Fabric community, I’m running a piece of code in a Fabric notebook that performs a few .collect() and .first() operations on a very small dataset (less than 5 columns). Intermittently, the j...
ati_puri
Resolver III
Resolver IIIHi,
Generally it is not advised to use .collect() even while working with small subset of data as this functions holds the result back to the driver memory. The interaction between the driver and workers while transferring the result makes it more slower as compared to other functions like .take() etc. Its is advisable to use .take(), filter by using .limit() function , cache or persist the results and clear spark cache for releasing driver memoery.
Thanks
v-aatheeque
Community Support
Community SupportHi navakanth_DE
Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.
- v-aatheeque
Hi navakanth_DE
We wanted to follow up to check if you’ve had an opportunity to review the previous responses. If you require further assistance, please don’t hesitate to let us know.