Random Forests SVM

Bagged classier using decision trees:

 1) Each split only considers a random group of features
2) Tree is grown to maximum size without pruning
3) Final predictions obtained by aggregating over the B trees


Out of Bags(OOB) samples:

 From each observation, construct its random forest predictor by averaging only those trees corresponding to bootstrap samples in which observation does not appear

 OOB error estimates can be fit in one sequence

 Once OOB stabilizes, training can be stopped

 OOB can also be used for variable importance

 Ensembles and Multi-Learners

 Goal: use multiple learners to solve parts of the same problem 
Ensembles: competing learners with multiple looks at the same problem

 SVM： Support Vector Machine(SVM)

Find large margin separator to improve generalization
Use optimization to find solution with few errors
Use kernel trick to make large feature spaces computationally efficient
Chose the linear separator with the largest margin
Robust to outliers

SVM unconstrained optimization problem:

 Equivalent form looks like regularization term + hinge loss
As C gets large, have to separate the data
As C gets small, ignores the data entirely

 Kernels:

Kernel function is used to make non-linear feature map
Think of kernel measure as similarity

 Polynominal SVM:

Example: polynomial of degree 2
Common Kernels:

 Kernel SVM: overfitting
Control overfitting by setting C via cross-validation, choose better kernel, vary parameters of the kernel(i.e., width of Gaussian )
Purpose of Validation:
1. performing model selection
2. avoid overfitting
3. selecting the right parameters

 Monte-Carlo Cross-Validation:
As know as random sub-sampling
Randomly select some fraction of your data to  form training set
Assign rest to test set
Repeat multiple times with new partitions

 Cross Validation must be applied to the entire sequence of modeling steps

Comments

Post a Comment

Popular posts from this blog

Reading CLIP

Learning directly from raw text about images is promising which leverages a much broader source of supervision. Task: predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of pairs. After pre-training, natural language is used to is used to reference learned visual concepts enabling zero-shot transfer of the model to downstream tasks.   [Paper Source]  https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021. Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020 .
Read more

Reading CutPaste

 CutPaste: a simple data augmentation strategy that cuts an image patch and pastes at a random location of a large image. Dataset: MVTec anomaly detection dataset Aim: detect various types of real-world defects [ Code ] Paper Source:  https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf
Read more

OOD-related papers

Deep One-Class Classification, ICLR 2021 [ Paper ] [ Code ] Google AI [ Blog ] [ Code ] Learning from Failure: Training Debiased Classifier from Biased Classifier [ Paper ] [ Code ] NeurIPS 2020 Learning Debiased Representation via Disentangled Feature Augmentation [ Paper ] [ Code ] NeurIPS 2021 Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization [ Paper ] Explainable Deep One-class Classification [ Paper ] GAN Ensemble for Anomaly Detection [ Paper ] Neural Transformation Learning for Deep Anomaly Detection Beyond Images [ Paper ] Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection Dual Compositional Learning in Interactive Image Retrieval [*] SSL  Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity [ Paper ] [ Code ] NeurIPS 2021 Unsupervised Learning of Discriminative Attributes and Visual Representations [ Paper ] CVPR2016 Common Visual Pattern Dis...
Read more