常见的激活函数

1. sigmoid 
函数定义为：
Sigmoid: exists between 0 to 1, it is especially used for models we have to predict the probability as an output.
The function is differentiable and monotonic, but the function's derivative is not, which can cause a neural network to get stuck at the training time.

Softmax function is a more generalized logistic activation function which is used for mutliclass classification.


2、tanh函数 (tangent activation function)

The range of tanh function is from -1 to 1.
The advantage is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph.

The function is also differentiable and monotonic while its derivative is not monotonic.

The tanh  function is mainly used in classification between two classes.

Both tanh and logistic sigmoid activation functions are used in feed-forward nets.

3. ReLU(Rectified Linear Unit) activation function

f(x) = max(0, x)
The ReLU  is half rectified. 
Range:[0, infinity)

The function and its derivative both are monotonic.

Problems: any negative input given to the ReLU activation function turns value to zero immediately in the graph.

4. Leaky ReLU: to solve the dying ReLU problem
Range: (-infinity, infinity)

Both Leaky and Randomized ReLU function and their derivatives are monotonic in nature.


Adopt from https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6

Comments

Post a Comment

Popular posts from this blog

Reading CLIP

Learning directly from raw text about images is promising which leverages a much broader source of supervision. Task: predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of pairs. After pre-training, natural language is used to is used to reference learned visual concepts enabling zero-shot transfer of the model to downstream tasks.   [Paper Source]  https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021. Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020 .
Read more

Reading CutPaste

 CutPaste: a simple data augmentation strategy that cuts an image patch and pastes at a random location of a large image. Dataset: MVTec anomaly detection dataset Aim: detect various types of real-world defects [ Code ] Paper Source:  https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf
Read more

OOD-related papers

Deep One-Class Classification, ICLR 2021 [ Paper ] [ Code ] Google AI [ Blog ] [ Code ] Learning from Failure: Training Debiased Classifier from Biased Classifier [ Paper ] [ Code ] NeurIPS 2020 Learning Debiased Representation via Disentangled Feature Augmentation [ Paper ] [ Code ] NeurIPS 2021 Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization [ Paper ] Explainable Deep One-class Classification [ Paper ] GAN Ensemble for Anomaly Detection [ Paper ] Neural Transformation Learning for Deep Anomaly Detection Beyond Images [ Paper ] Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection Dual Compositional Learning in Interactive Image Retrieval [*] SSL  Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity [ Paper ] [ Code ] NeurIPS 2021 Unsupervised Learning of Discriminative Attributes and Visual Representations [ Paper ] CVPR2016 Common Visual Pattern Dis...
Read more