Microsoft fabrics - Default Environment - Ml Model - Connection Refused error

Hi ca_solution 

You're encountering a Connection refused error during the execution of your PySpark code, which uses SynapseML's LightGBM classifier for hyperparameter tuning within a Spark environment. This error is not due to a coding mistake, but rather a network-level issue within the Spark cluster. Specifically, SynapseML's LightGBM component relies on inter-node communication over TCP ports to coordinate training across distributed workers. If those ports are blocked, unavailable, or misconfigured, the Spark tasks will fail with a java.net.ConnectException, indicating they cannot reach each other. This can happen if the Spark environment (e.g., Azure Synapse, Databricks, HDInsight) has firewall restrictions, lacks sufficient resources (like CPU or memory), or is misconfigured for distributed LightGBM usage. It can also occur if too many parallel tasks are launched at once—exceeding what the cluster can handle—especially during cross-validation. To resolve this, you should ensure that the environment allows node-to-node communication, possibly by opening necessary ports (typically starting at 12400), reducing the level of parallelism, and reviewing Spark cluster logs for more details. Ultimately, this is an infrastructure or environment issue that affects how the LightGBM workers connect across the distributed Spark cluster, and addressing it requires adjusting networking or resource configurations in your cluster setup.