Confusion Matrix

• Sensitivity / true positive rate (TPR) / hit rate / recall
  • $$\frac{TP}{P} = \frac{TP}{TP + FN}$$
• Specificity (SPC) / true negative rate (TNR)
  • $$\frac{TN}{N} = \frac{TN}{TN + FP}$$
• Precision / positive predictive value (PPV)
  • $$\frac{TP}{TP + FP}$$
• Negative predictive value (NPV)
  • $$\frac{TN}{TN + FN}$$
• Fall-out / false positive rate (FPR)
  • $$\frac{FP}{N} = \frac{FP}{FP + TN} = 1 - SPC$$
• False negative rate (FNR)
  • $$\frac{FN}{TP+FN} = 1 - TPR$$
• False discovery rate (FDR)
  • $$\frac{FP}{TP+FP} = 1-PPV$$

Precision/Recall

• Precision
  • When it says true, the ratio that it is correct
• Recall
  • The ratio the truths are detected
• F1
  • Harmonic mean of precision & recall
  • $$F_1 = 2 \frac{\text{precision} \text{recall}}{\text{precision} + \text{recall}}$$

Tradeoff

Increasing precision reduces recall, and vice versa.

Receiver Operating Characteristic (ROC)

• True positive rate
  • = recall = hit rate = sensitivity = prob. of detection
• False positive rate
  • = fall-out = prob. of false alarm = 1 - specificity (true negative rate)
• Area under ROC curve (AUC)
  • 1 for perfect classifier
  • 0.5 for random

Tradeoff

Increasing TPR also increases FPR.

Comparison

• PR
  • If positive class rare
  • Care more about false positives than false negatives
• ROC
  • Otherwise

Multiclass Classification

• One-vs-All (OvA)
  • Train K linear classifiers, one for each class
  • The one with highest score wins
• One-vs-One (OvO)
  • Train $$\frac{K(K-1)}{2}$$ classifiers, one for each pair of classes, trained with relevant samples only
  • SVM default, due to space constraints