Clustering vs Dimensionality Reduction

Clustering and Dimensionality Reduction are applied to the problem of unsupervised learning.
Clustering identifies unknown structure in the data; 
Dimensionality Reduction uses structural characteristics to simplify data.
However, there is a problem called Curse of Dimensionality.
In practice, too many features leads to worse performance.
Therefore, we need Dimensionality Reduction.

It's possible to represent data with fewer dimensions, which requires us to discover intrinsic dimensionality of data.

One way to do dimensionality reduction is to perform lower dimensional projections.

In the way, we could transform dataset to have less features, in the new feature space, some original features are combined via linear or nonlinear functions.

PCA: Principal Component Analysis

Find sequence of linear combinations of the features that have maximal variance and are uncorrelated

PCA:1st PC

1st PC of X is unit vector that maximizes the sample variance compared to all other unit vectors
v1=argmax||v1||=1(Xv)^T(Xv)

 1st PC score: Xv1

PCA:Next PC 

Idea: Successively find orthogonal directions of highest variance

The reason of Orthogonal:
1) want to minimize reduancy
2) want to look at variance in different directions
3) computation is easier

Basic PCA Algorithm

1) start with a zero-centered mxn data matrix X
2) compute covariance matrix
3) find eigenvectors of covariance matrix
4) PCs: k eigenvectors with highest eigenvalues

PCA: In practice

Forming the covariance matrix can require a lot of memory, especially when number of sampels>> numbers of features

Typical approach: use singular value decomposition (SVD)

SVD: each matrix can be decomposed using singular value decomposition

PCA: Applications

~Visualizing high-dimensional data
~Finding hidden relationships
~Compressing information
~Avoiding redundancy
~Detecting outliers/denosining
~Reducing model complexity

The difference between PCA and LDA

PCA: reduce dimensionality while preserving as much of the variance in high-dimensional space as possible

LDA: reduce dimensionality while preserving as much of the class discriminatory information as possible

Comments

Post a Comment

Popular posts from this blog

Reading CLIP

Learning directly from raw text about images is promising which leverages a much broader source of supervision. Task: predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of pairs. After pre-training, natural language is used to is used to reference learned visual concepts enabling zero-shot transfer of the model to downstream tasks.   [Paper Source]  https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021. Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020 .
Read more

Reading CutPaste

 CutPaste: a simple data augmentation strategy that cuts an image patch and pastes at a random location of a large image. Dataset: MVTec anomaly detection dataset Aim: detect various types of real-world defects [ Code ] Paper Source:  https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf
Read more

OOD-related papers

Deep One-Class Classification, ICLR 2021 [ Paper ] [ Code ] Google AI [ Blog ] [ Code ] Learning from Failure: Training Debiased Classifier from Biased Classifier [ Paper ] [ Code ] NeurIPS 2020 Learning Debiased Representation via Disentangled Feature Augmentation [ Paper ] [ Code ] NeurIPS 2021 Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization [ Paper ] Explainable Deep One-class Classification [ Paper ] GAN Ensemble for Anomaly Detection [ Paper ] Neural Transformation Learning for Deep Anomaly Detection Beyond Images [ Paper ] Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection Dual Compositional Learning in Interactive Image Retrieval [*] SSL  Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity [ Paper ] [ Code ] NeurIPS 2021 Unsupervised Learning of Discriminative Attributes and Visual Representations [ Paper ] CVPR2016 Common Visual Pattern Dis...
Read more