Ensemble Methods in Machine Learning

Ensemble Methods:

Ensemble Methods combines several decision trees to produce better predictive performance than utilizing a single decision tree. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner. Ensemble Methods can be divided into groups:

1. Sequential Ensemble Methods:

These ensemble methods where the base learners are generated sequentially. The basic motivation of sequential methods is to exploit the dependence between the base learners. The overall performance can be boosted by weighing previously mislabeled examples with higher weight.

2. Parallel Ensemble Methods:

These ensemble methods where the base learners are generated in parallel. The basic motivation of parallel methods is to exploit independence between the base learners since the error can be reduced dramatically by averaging.

Types of Ensemble Methods:

Ensemble Methods are meta-algorithms that combine several machine learning techniques into one predictive model in order to:

Ensemble Methods in Machine Learning

    1. Decrease variance (Bagging)
    2. Bias (Boosting)
    3. Improve predictions (Stacking)

Bagging:

Bagging involves fitting many decision trees on different samples of the same dataset and averaging the predictions.

Boosting:

Boosting involves adding ensemble members sequentially that correct the predictions made by prior models and outputs a weighted average of the predictions.

Stacking:

Stacking involves fitting many different models types on the same data and using another model to learn how to best combine the predictions.