The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hi
We are currently running approximately 50 tables dynamically with a single notebook in a pipeline that merge into lakehouse tables. We did not specify any batch count, and we noticed that 20 notebooks were running concurrently. However, we've been encountering an intermittent issue while executing cells in the middle of the notebook. Could someone kindly assist us with troubleshooting this problem?
Could this be related to the Livy session expiring during notebook execution, or is it potentially caused by something else
Error:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (jobservice.eastus.trident.azuresynapse.net:443) --> SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Below are the error logs i found from the monitor tab:
Solved! Go to Solution.
Hi @SivaReddy24680,
Thank you for reaching out in Microsoft Community Forum.
The issue may be caused by Livy session timeouts, high concurrency, or resource limitations. please follow below steps to resolve the error;
1. Adjust settings or add a keep-alive command to prevent session expiration.
2. Set a batch size in your pipeline to limit parallel execution and avoid resource exhaustion.
3. Make sure Ensure there are no firewall/proxy restrictions blocking Azure Synapse connections.
4. High workloads may exceed CPU/memory limits, so consider scaling up your Spark cluster.
Please continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.
Hi @Anonymous
we reduced the concurrency of the notebooks. As the issue is intermittent earlier, we will monitor for few days to make sure the issue is resolved
Hi @SivaReddy24680,
I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Please continue using Microsoft community forum.
Thank you,
Pavan.
Hi @SivaReddy24680,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept as Solution" and give a 'Kudos' so other members can easily find it.
Thank you,
Pavan.
Hi @Anonymous
we reduced the concurrency of the notebooks. As the issue is intermittent earlier, we will monitor for few days to make sure the issue is resolved
Hi @SivaReddy24680,
Thank you for reaching out on the Microsoft Community Forum.
We hope your issue has been resolved. If you have any further questions, please feel free to contact us. If your issue is resolved, please "Accept as Solution" so that other community members can find the solution quickly.
Please continue using the Microsoft Community Forum.
Regards,
Pavan.
Hi @SivaReddy24680,
Thank you for reaching out on the Microsoft Community Forum.
We hope your issue has been resolved. If you have any further questions, please feel free to contact us. If your issue is resolved, please "Accept as solution" so that other community members can find the solution quickly.
Please continue using the Microsoft Community Forum.
Regards,
Pavan.
Hi @SivaReddy24680,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, kindly "Accept as Solution" and give it a 'Kudos' so others can find it easily.
Thank you,
Pavan.
Hi @SivaReddy24680,
Thank you for reaching out in Microsoft Community Forum.
The issue may be caused by Livy session timeouts, high concurrency, or resource limitations. please follow below steps to resolve the error;
1. Adjust settings or add a keep-alive command to prevent session expiration.
2. Set a batch size in your pipeline to limit parallel execution and avoid resource exhaustion.
3. Make sure Ensure there are no firewall/proxy restrictions blocking Azure Synapse connections.
4. High workloads may exceed CPU/memory limits, so consider scaling up your Spark cluster.
Please continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.
Hi
We are currently running approximately 50 tables dynamically with a single notebook in a fabric pipeline that merge the data into the lakehouse tables. We did not specify any batch count, and we noticed that 20 notebooks were running concurrently. However, we've been encountering an intermittent issue while executing cells in the middle of the notebook. Could someone kindly assist us with troubleshooting this problem?
Could this be related to the Livy session expiring during notebook execution, or is it potentially caused by something else?
Error:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (jobservice.eastus.trident.azuresynapse.net:443) --> SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Logs:
User | Count |
---|---|
20 | |
17 | |
6 | |
2 | |
2 |
User | Count |
---|---|
51 | |
49 | |
17 | |
6 | |
4 |