BatchSize、 学习率 、 Data augmentation、 Internal Covariate Shift

Batchsize 增大，达到同样精度所需要的epoch数量变多，因为随着样本数量的增大，标准差并不是线性减小的。

 Batchsize 大更容易陷入局部最优，在相同的计算量下，因为标准差的问题收敛速度更慢。

Batchsize增大到某个时候就会达到时间上的最优。

Data Augmentation:  


Color Jittering: 图像亮度，饱和度，对比度  
PCA Jittering: 按照RGB 三个颜色通道计算均值和标准差，再在整个训练集上计算协方差矩阵，进行特征分解得到特征向量和特征值，用来做PCA Jittering
Random Scale
Random Crop
Horizontal/Vertical Flip: 水平、垂直翻转
Shift: 平移变换
Rotation/Reflection: 旋转、仿射变换
Noise： 高斯噪声、模糊处理
Label Shuffle

Internal Covariate Shift: 


深度神经网络涉及到很多层的叠加，每一层参数更新都会导致上层输入数据分布发生变化，通过层层叠加，高层的输入分布会变化非常剧烈。 Covarite shift 就是值source domain 和target dmain 的条件概率是一致的，但边缘概率不同。

 ICS 会导致的问题：每个神经元的输入数据不再是独立同分布的。
1） 上层参数需要不断适应新的输入数据分布，降低学习速度
2） 下层输入的变化可能趋向于变大或者变小，导致上层落入饱和区，使得学习过早停止
3） 每层的更新都会影响到其它层，因此每层的参数更新策略需要尽可能的谨慎。

Comments

Post a Comment

Popular posts from this blog

Reading CLIP

Learning directly from raw text about images is promising which leverages a much broader source of supervision. Task: predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of pairs. After pre-training, natural language is used to is used to reference learned visual concepts enabling zero-shot transfer of the model to downstream tasks.   [Paper Source]  https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021. Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020 .
Read more

Reading CutPaste

 CutPaste: a simple data augmentation strategy that cuts an image patch and pastes at a random location of a large image. Dataset: MVTec anomaly detection dataset Aim: detect various types of real-world defects [ Code ] Paper Source:  https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf
Read more

OOD-related papers

Deep One-Class Classification, ICLR 2021 [ Paper ] [ Code ] Google AI [ Blog ] [ Code ] Learning from Failure: Training Debiased Classifier from Biased Classifier [ Paper ] [ Code ] NeurIPS 2020 Learning Debiased Representation via Disentangled Feature Augmentation [ Paper ] [ Code ] NeurIPS 2021 Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization [ Paper ] Explainable Deep One-class Classification [ Paper ] GAN Ensemble for Anomaly Detection [ Paper ] Neural Transformation Learning for Deep Anomaly Detection Beyond Images [ Paper ] Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection Dual Compositional Learning in Interactive Image Retrieval [*] SSL  Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity [ Paper ] [ Code ] NeurIPS 2021 Unsupervised Learning of Discriminative Attributes and Visual Representations [ Paper ] CVPR2016 Common Visual Pattern Dis...
Read more