Ensemble Methods in Machine Learning

Ensemble Methods:

Ensemble Methods are meta-algorithms that combine several machine learning techniques into one predictive model in order to:

    1. Decrease variance (Bagging)
    2. Bias (Boosting)
    3. Improve predictions (Stacking)

Ensemble Methods combines several decision trees to produce better predictive performance than utilizing a single decision tree. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner.

Ensemble Methods can be divided into groups:

1. Sequential Ensemble Methods:

These ensemble methods where the base learners are generated sequentially. The basic motivation of sequential methods is to exploit the dependence between the base learners. The overall performance can be boosted by weighing previously mislabeled examples with higher weight.

2. Parallel Ensemble Methods:

These ensemble methods where the base learners are generated in parallel. The basic motivation of parallel methods is to exploit independence between the base learners since the error can be reduced dramatically by averaging.