Forum Discussion
How do you decide which ML algorithm to start with for a new dataset?
Hi richa_gupta_224 ,
In real-world ML projects, we usually don’t try to identify the “best” algorithm upfront. Instead, the focus is on understanding the data and establishing baselines.
A typical approach looks like this:
Start with a simple baseline (Linear or Logistic Regression). This helps validate the data, uncover leakage, and set a reference point. Evaluate a few strong models in parallel, usually one linear model, one tree-based model (Random Forest), and one boosting model (XGBoost/LightGBM). The data often makes the choice clearer.
Let practical constraints guide selection:
Need interpretability > simpler models
Tabular data and performance-driven > tree-based or boosting models
Small datasets > avoid overly complex models
Be cautious of extremely high accuracy. Iterate over time. As data, features, or business requirements evolve, the “best” model can change.
The key takeaway is that model selection is an iterative engineering process, not a one-time decision. In most cases, improvements come more from better data and features than from switching algorithms.
Hope this helps.
Thank you.
Hi richa_gupta_224 ,
In real-world ML projects, we usually don’t try to identify the “best” algorithm upfront. Instead, the focus is on understanding the data and establishing baselines.
A typical approach looks like this:
Start with a simple baseline (Linear or Logistic Regression). This helps validate the data, uncover leakage, and set a reference point. Evaluate a few strong models in parallel, usually one linear model, one tree-based model (Random Forest), and one boosting model (XGBoost/LightGBM). The data often makes the choice clearer.
Let practical constraints guide selection:
Need interpretability > simpler models
Tabular data and performance-driven > tree-based or boosting models
Small datasets > avoid overly complex models
Be cautious of extremely high accuracy. Iterate over time. As data, features, or business requirements evolve, the “best” model can change.
The key takeaway is that model selection is an iterative engineering process, not a one-time decision. In most cases, improvements come more from better data and features than from switching algorithms.
Hope this helps.
Thank you.
Thank you for the clear explanation. I really liked the idea of starting with simple baselines and letting data and constraints guide the choice, it makes the process feel much more practical than chasing the best algorithm upfront.
I had one follow-up question that When setting a baseline, how do you decide if the model is good enough to move forward, or if the issue lies more in feature engineering rather than trying a more complex model?Hi richa_gupta_224 ,
Thank you for the update.
Build a simple model like Linear Regression, Logistic Regression, or a simple Decision Tree. Evaluate it using appropriate metrics (e.g., accuracy, F1, RMSE, ROC-AUC). This gives you a reference point.
If your baseline model performs poorly even on training data, this usually indicates features are weak or relationships are not captured, not just algorithm choice.
Example: Training RMSE is high > likely missing key features or non-linear patterns.
High training performance but low validation performance > model overfitting; complex models might help, but so can regularization or more data.
Both training and validation low > features may be insufficient; try feature engineering first.Check feature importance (tree-based models) or coefficients (linear models) to see if features have signal.
Do EDA: correlations, distributions, missing values, categorical encoding.
Sometimes creating new features or transforming existing ones gives more improvement than swapping algorithms.If baseline is reasonable and interpretable, but business requirements need higher accuracy, try tree-based or boosting models. If baseline is poor, invest in better feature engineering, handling missing data, or collecting more data before chasing complex algorithms.
Hope this helps.
Best Regards.
