Stacking

Basic Idea

The basic idea behind stacked generalization is to use a pool of base classifiers, then use another classifier to combine their predictions, with the aim of reducing the generalization error.

Algorithms

Standard Stacking

• Split training set (labels,F)$(labels,F)$ into k$k$ folds f1,f2,…,fk$f_1,f_2,\dots ,f_k$, fit learner Li$L_i$ on each k−1$k-1$ folds and use it to predict the remaining fold to get predictions on f1,f2,…,fk$f_1,f_2,\dots ,f_k$, denoted as Vi1,Vi2,…,Vik$V_{i1},V_{i2},\dots ,V_{ik}$, one fold at a time.
• Meanwhile, each time we get a Li$L_i$, use it to make predictions on the whole test set thus we obtain k$k$ sets of predictions Pi1,Pi2,…,Pik$P_{i1},P_{i2},\dots ,P_{ik}$.
• Vi=(Vi1;Vi2;…;Vik)$V_i=(V_{i1};V_{i2};\dots ;V_{ik})$
  Pi=1k∑km=1Pim$P_i=\frac{1}{k}\sum _{m=1}^k{P}_{im}$ or other averaging methods
• Repeat above steps from i=1$i=1$ to n$n$, using different learners which we call unified as Model 1, we have
  V=(V1,V2,…,Vn)$V=(V_1,V_2,\dots ,V_n)$
  P=(P1,P2,…,Pn)$P=(P_1,P_2,\dots ,P_n)$
• Consider (labels,V)$(labels,V)$ and P$P$ as new training set and test set respectively, train and make predictions with Model 2 (usually Logistic Regression, Popular non-linear algorithms for stacking are GBM, KNN, NN ,RF**and **ET (extra trees).) to get final results.

In a word, it models

where wi$w_i$ is the weight of the i-th learner and gi$g_i$ is the corresponding prediction. Some details should be paid attention to:

• Averaging works if it is a regression problem or it is a classification problem but model 1 learners’ output are probabilities. Under other circumstances, voting could be better than averaging.
• In fact you can also get Pi$P_i$ by simply using Li$L_i$ to train on the whole training set and make predictions on the test set , which may consume more computing resources but slightly lower the coding complexity.
• The partitions of training set must be the same for n$n$ different estimators, especially when you’re on a team work, or it will lead to information leak and therefore over-fitting.

Feature-Weighted Linear Stacking

Replace wi$w_i$ with ∑vkxk$\sum v_k{x}_k$, here xk$x_k$ represents the k-th feature of a sample and vk$v_k$ is the corresponding weight.

Further

We can also insert predictions of Model 1 to original features to get expanded features, notice that their dimensions are different, therefore normalization is necessary.

References

• Stacked Generalization
  http://www.machine-learning.martinsewell.com/ensembles/stacking/Wolpert1992.pdf
• Feature-Weighted Linear Stacking
  https://arxiv.org/pdf/0911.0460.pdf
• Kaggle Ensemble Guide
  https://mlwave.com/kaggle-ensembling-guide/
• A Kaggler’s Guide to Model Stacking in Practice
  http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/
• 详解stacking过程
  https://blog.csdn.net/wstcjf/article/details/77989963
• Stacking Learning在分类问题中的使用 (with more valuable links)
  https://blog.csdn.net/MrLevo520/article/details/78161590
• 详解 Stacking 的 python 实现
  https://www.jianshu.com/p/5905f19c4df6