Sophie's Daily Note

 Goal: improve the worst-group performance without training group annotations Methods: first identify training examples that are misclassified by a standard ERM model; and then train the final model by up-weighting the examples identified in the first stage  

Background: Neural networks fail to perform well on certain groups of the data. The group information may be expensive to obtain.  Previous work: improve worst-group performance even when group labels are unavailable for robustness and fairness Problem: improve group robustness when only some group labels are available  Methods: a two-step framework to utilize the partial labels for training data and then use the predicted group labels in a robust optimization objective Keywords:  DRO - Distributionally Robust Optimization GDRO - Group Distributionally Robust Optimization

 CutPaste: a simple data augmentation strategy that cuts an image patch and pastes at a random location of a large image. Dataset: MVTec anomaly detection dataset Aim: detect various types of real-world defects [ Code ] Paper Source:  https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf

 Deep One-class Classification: a two-stage framework for deep one-class classification. 1) learn self-supervised representations from one-class data 2) build one-class classifiers on learned representations Self-supervised representation algorithms, the connections to existing one-class classification methods, SOTA contrastive representation learning for one-class classification A: the stochastic data augmentation process, including resize and crop, horizontal flip, color jittering, gray-scale and gaussian blur. Feature extractor f, loss L, projection head g. network structure as gof, g is the projection head used to compute proxy losses and f outputs representations for the downstream task. One way of representation learning is to learn by discriminating augmentations applied to the data. Although not trained to do so, the likelihood of learned rotation classifiers is shown to well approximate the normality score and has been used for one-class classification. A plausible ex...

Deep One-Class Classification, ICLR 2021 [ Paper ] [ Code ] Google AI [ Blog ] [ Code ] Learning from Failure: Training Debiased Classifier from Biased Classifier [ Paper ] [ Code ] NeurIPS 2020 Learning Debiased Representation via Disentangled Feature Augmentation [ Paper ] [ Code ] NeurIPS 2021 Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization [ Paper ] Explainable Deep One-class Classification [ Paper ] GAN Ensemble for Anomaly Detection [ Paper ] Neural Transformation Learning for Deep Anomaly Detection Beyond Images [ Paper ] Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection Dual Compositional Learning in Interactive Image Retrieval [*] SSL  Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity [ Paper ] [ Code ] NeurIPS 2021 Unsupervised Learning of Discriminative Attributes and Visual Representations [ Paper ] CVPR2016 Common Visual Pattern Dis...

VDVAE:  a hierarchical VAE, generate samples quickly and outperform the PixelCNN in log-likelihood on all the natural image benchmarks. In theory, VAEs can actually represent autoregressive models. VAEs can learn first generate global features at low resolution, then fill in local details in parallel at higher resolutions. Many types of generative models have flourished in recent years, including likelihood-based generative models, which include autoregressive models, VAEs, and invertible flows. Their objective, the negative log-likelihood, is equivalent to the KL divergence between the data distribution and the model distribution.  1) provide theoretical justification for why greater depth could improve VAE performance 2) introduce an architecture capable of scaling past 70 layers 3) verify that depth, independent of model capacity, improves log-likelihood, and allows VAEs to outperform the PixelCNN on all benchmarks 4) uses fewer parameters, generates sam...

  TransFG:       1)  verify the effectiveness of vision transformer on fine-grained visual classification which offers an alternative to the dominating CNN backbone with RPN model design     2) naturally focuses on the most discriminative regions of the objects and achieve SOTA performance      3) visualization helps show the ability of capturing discriminative image regions Methods:     1) vision transformer as feature extractor          image sequentialization: first preprocess the input image into a sequence of flattened patches, generating overlapping patches with sliding window     2) TransFG architecture, propose the Part Selection Module (PSM) and apply contrastive feature learning to enlarge the distance of representations between similar sub-categories     3) contrastive feature learning, minimizes the similarity of classification tokens corresponding to diffe...

Learning directly from raw text about images is promising which leverages a much broader source of supervision. Task: predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of pairs. After pre-training, natural language is used to is used to reference learned visual concepts enabling zero-shot transfer of the model to downstream tasks.   [Paper Source]  https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021. Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020 .

Adapt from  https://www.jeremyjordan.me/semantic-segmentation/ The goal of semantic image segmentation is to label each pixel of an image with a corresponding class of what is being represented. Because we're predicting for each pixel in the image, this task is commonly referred to as dense prediction. Earlier layers tend to learn low-level concepts while later layers develop more high-level feature mappings. Drozdzal et al.   swap out the basic stacked convolution blocks in favor of residual blocks. This residual block introduces short skip connections alongside the existing long skip connections between the corresponding feature maps of encoder and decoder modules found in the standard U-Net structure. The report that the short skip connections allow for faster convergence when training and allow for deeper models to be trained. Expanding on this, Jegou et al. proposed the use of dense blocks, still following  a U-Net structure,  arguing that the "characte...

Paper from: https://arxiv.org/pdf/2004.10956.pdf Incremental learning new classes Few-shot class-incremental learning problem(FSCIL) FSCIL requires CNN models to incrementally learn new classes from very few labelled samples. Topology-perserving knowledge incrementer framework(TOPIC) To mitigate forgetting, most class-incremental learning(CIL) works use the knowledge distillation technique that maintains the network's output logins correspond to old classes. They usually store s set of old class exemplars and apply the distillation loss to the network's output. Problems: 1)class-imbalance problem; 2)performance trade-off between old and new classes TOPIC uses a neural gas(NG) network to model the topology of feature space.  When learning the new classes, NG grows to adapt to the change of feature space. On this basis, we formulate FSCIL as an optimization problem with two objectives.  1) to avoid catastrophic forgetting, TOPIC preserves the old...

Java 注解： Java 语言中的类、方法、变量、参数和包都可以被标注。 Java标注可以通过反射获取标注内容。 内置的注解： Java定义了一套注解，共有7个，3个在java.lang中，剩下的4个在java.lang.annotation中。 @override--检查该方法是否是重写方法。如果发现其父类，或者引用的接口并没有该方法时，会报编译警告。 @Deprecated--标记过时方法。如果使用该方法，会报编译警告。 @SupperessWarning--指示编译器去忽略注解中声明的警告。 @Retention--标识这个注解应该怎么保存，是只在代码中，还是编入class文件中，或者是在运行时间可以通过反射访问。 @Documented--标记这些注解是否包含在用户文档中。 @Target--标记这个注解应该是哪种Java成员。 @Inherited--标记这个注解是继承于哪个注解类。

# This problem was asked by Alibaba. # Given an even number (greater than 2), return two prime numbers whose sum will be equal to the given number. def is_prime(n, primes):     for p in primes:         if n == p:             return True         if not n%p == 0:             return False     return True def get_primes(num):     limit = num//2     primes = list()     candidates = list()     for i in range(2, limit+1):         if is_prime(i, primes):             primes.append(i)             candidates.append((i, num-i))     new_candidates = list()     for p1, p2 in candidates[::-1]:         if is_prime(p2, primes):             primes.append(p2)     ...

Image similarity: the measure of how similar two images are. In other words, it quantifies the degree of similarity between intensity patterns in two images. Triplet: a triplet contains a query image, a positive image and a negative image. How to measure the similarity of two images? L1-norm: Manhattan distance L2-norm: Euclidean distance Loss function: The whole deep ranking architecture can be thought of as a function that would map the image to a point in the Euclidean space. The goal is to learn an embedding function that assigns smaller distance to more similar images. D(f(pi), f(pi+)) < D(f(pi), f(pi-)) <pi, pi+, pi-> such that r(pi, pi+) > r(pi,pi-) f is the embedding function that would map the image to a vector. pi is the query image, pi+ is the positive image, pi- is the negative image and r is the similarity distance between two images. The hinge loss for the triplet is defined as: l(pi, pi+, pi-) = max{0, g+D(f(pi), f(pi+)) - D(f(pi),f(pi-...

zip: returns a list of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. a = [1, 2, 3] b = ['a', 'b', 'c'] for i, j in zip(a, b):      print(i,j) =>1 a =>2 b =>3 c Python __repr__() function returns the object representation. It could be any valid python expression such as tuple, dictionary, string etc.  id(object): return the "identity" of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same id() value. hasattr() The hasattr() method returns true if an object has the given named attribute and false if it does not. hasattr(object, name) PIL. Image.resize(size, resample = 0) returns a resized copy of this image Parameters:  -size: the requested size in pixels, as a 2-tuple -resample: an optional resampling filter. This can be one of PIL.I...

torch.nn.functioanl.adaptive_avg_pool2d(input, output_size) Applies a 2D adaptive average pooling over an input signal composed of several input planes. torch.nn.functional.adaptive_avg_pool3d(input, output_size) Applies a 3D adaptive average pooling over an input signal composed of several input planes Class: torch.nn.Parameter A kind of Tensor that is to be considered a module parameter. parameters: data(Tensor) torch.nn.functional.interpolate(input, size, scale_factor, mode= 'nearest', align_corners) Down/up samples the input to either the given size or the given scale_factor. The algorithm used for interpolation is determined by mode. torch.cat(tensors, dim = 0, out = None) concatenates the given of seq tensors in the given dimension. All tensors must be either have the same shape or be empty.

Bagged classier using decision trees: 1) Each split only considers a random group of features 2) Tree is grown to maximum size without pruning 3) Final predictions obtained by aggregating over the B trees Out of Bags(OOB) samples: From each observation, construct its random forest predictor by averaging only those trees corresponding to bootstrap samples in which observation does not appear OOB error estimates can be fit in one sequence Once OOB stabilizes, training can be stopped OOB can also be used for variable importance Ensembles and Multi-Learners Goal: use multiple learners to solve parts of the same problem  Ensembles: competing learners with multiple looks at the same problem SVM： Support Vector Machine(SVM) Find large margin separator to improve generalization Use optimization to find solution with few errors Use kernel trick to make large feature spaces computationally efficient Chose the linear separator ...

（1 ） 当分类边界是线性时， LR 和 LDA 会更好; 当分类边界是非线性时， QAD 会更好； 当分类边界更复杂时， KNN 会更好。 (2) LR 和 LDA 都将产生线性分类边界，不同的是LR的系数估计是通过极大似然法， 而LDA系数是运用正态分布的均值和方差的估计值计算的， LR 适用于二分类问题，对于多分类问题，LDA 更为常见。 LDA 和 QDA 都是建立在自变量服从正态分布的假设上，所以当自变量的分布确实是几乎服从正态分布时，这两种方法表现的较好； LDA 和QDA 的区别在于LDA 假设所有类别的自变量都服从方差相同的正态分布，而QDA假设对于因变量属于不同类别的自变量服从方差不同的正态分布，选择LDA和QDA的关键在于bias-variance的权衡。

Batchsize 增大，达到同样精度所需要的epoch数量变多，因为随着样本数量的增大，标准差并不是线性减小的。 Batchsize 大更容易陷入局部最优，在相同的计算量下，因为标准差的问题收敛速度更慢。 Batchsize增大到某个时候就会达到时间上的最优。 Data Augmentation:   Color Jittering: 图像亮度，饱和度，对比度    PCA Jittering: 按照RGB 三个颜色通道计算均值和标准差，再在整个训练集上计算协方差矩阵，进行特征分解得到特征向量和特征值，用来做PCA Jittering Random Scale Random Crop Horizontal/Vertical Flip : 水平、垂直翻转 Shift: 平移变换 Rotation/Reflection: 旋转、仿射变换 Noise ： 高斯噪声、模糊处理 Label Shuffle Internal Covariate Shift:  深度神经网络涉及到很多层的叠加，每一层参数更新都会导致上层输入数据分布发生变化，通过层层叠加，高层的输入分布会变化非常剧烈。 Covarite shift 就是值source domain 和target dmain 的条件概率是一致的，但边缘概率不同。 ICS 会导致的问题 ：每个神经元的输入数据不再是独立同分布的。 1） 上层参数需要不断适应新的输入数据分布，降低学习速度 2） 下层输入的变化可能趋向于变大或者变小，导致上层落入饱和区，使得学习过早停止 3） 每层的更新都会影响到其它层，因此每层的参数更新策略需要尽可能的谨慎。

1. sigmoid  函数定义为： Sigmoid: exists between 0 to 1 , it is especially used for models we have to predict the probability as an output. The function is differentiable and monotonic , but the function's derivative is not, which can cause a neural network to get stuck at the training time. Softmax function is a more generalized logistic activation function which is used for mutliclass classification. 2、tanh函数 (tangent activation function) The range of tanh function is from -1 to 1. The advantage is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph. The function is also differentiable and monotonic while its derivative is not monotonic . The tanh  function is mainly used in classification between two classes . Both tanh and logistic sigmoid activation functions are used in feed-forward nets. 3. ReLU(Rectified Linear Unit) activation function f(x) = max(0, x) ...

Pooling 层作用：   1.invariance(translation, rotation, scale) 2. 保留主要特征的同事减少参数和计算量（降维），提高模型的泛化能力 在神经网络中，池化函数一般在卷积函数下一层 池化操作是利用一个矩阵窗口在张量上进行扫描，将每个矩阵窗口中的值通过取最大值或者平均值来减少元素个数。 avg_pool, max_pool, max_pool_with_argmax 尺度不变性增大了感受野， 让卷积看到更多的信息，但是增大感受野的前提是降低分辨率，丢失了重要信息，这对segmentation要求的精确度location有一定的影响。 但为了增大感受野一开始就用跟图像一样大的卷积核，这样感受野不就变大了吗？ 错，卷积层越深模型的表征能力越强，但此时降维会导致重要信息丢失。 Adopt from  https://blog.csdn.net/zxyhhjs2017/article/details/78607469