Knowledge Distillation 笔记

Inter-Region Affinity Distillation for Road Marking Segmentation （2020.04）

用 inter-region affinity graph 描述 structural relationship。
每个node表示一种class的 areas of interest (AOI)（同类算一个 or instance算一个？），edge 代表 Affinity

Generation of AOI：smooth the label map with an average kernel φ and AOI map as 
AOI-grounded moment pooling：分别描述 mean, variance, skewness

Inter-region affinity ：

distillation：

     

Experiments

A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning （CVPPR 2017）

用网络 layer 间的相似性作为蒸馏知识（Gramian matrix）
The knowledge transfer performance is very sensitive to how the distilled knowledge is defined. we believe that demonstrating the solution process for the problem provides better generalization than teaching the intermediate result
论文通过实验证明蒸馏三个好处：
1、Fast optimization
2、Performance improvement for the small DNN
3、Transfer Learning

训练过程：