Accuracy: $\frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$ (Overall correctness)

Precision: $\frac{\text{TP}}{\text{TP} + \text{FP}}$ (Proportion of positive identifications that were actually correct)

Recall (Sensitivity): $\frac{\text{TP}}{\text{TP} + \text{FN}}$ (Proportion of actual positives identified correctly)

F1-Score: $2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$ (Harmonic mean of precision and recall)

ROC Curve & AUC: Plot of True Positive Rate vs. False Positive Rate at various threshold settings. AUC (Area Under Curve) measures overall model performance.

5.3 Model Selection

Bias-Variance Trade-off:
- Bias: Error from erroneous assumptions in the learning algorithm (underfitting).
- Variance: Error from sensitivity to small fluctuations in the training set (overfitting).
Cross-Validation:
- Hold-out: Split data into training and testing sets.
- K-Fold CV: Data split into $k$ folds. Model trained $k$ times, each time using a different fold as test set.
- Leave-One-Out CV (LOOCV): $k=n$ (number of data points).
Hyperparameter Tuning:
- Grid Search: Exhaustively searches through a specified subset of hyperparameters.
- Random Search: Randomly samples hyperparameters from a given distribution.

6. Ensemble Methods

Combine multiple learning algorithms to obtain better predictive performance.
Bagging (Bootstrap Aggregating): Trains multiple models independently on different bootstrap samples of the training data.
- Random Forest: Ensemble of decision trees trained via bagging. Introduces randomness in feature selection.
Boosting: Sequentially builds models, where each new model attempts to correct errors of the previous ones.
- AdaBoost (Adaptive Boosting): Weights misclassified samples more heavily in subsequent iterations.
- Gradient Boosting: Builds models by minimizing a loss function using gradient descent.
- XGBoost, LightGBM, CatBoost: Optimized gradient boosting frameworks.
Stacking: Trains a meta-learner to combine predictions from multiple base models.

7. Decision Trees

Tree-like model where each internal node represents a "test" on an attribute, each branch represents an outcome of the test, and each leaf node represents a class label or a value.
ID3, C4.5, CART: Common algorithms for building decision trees.
Splitting Criteria:
- Gini Impurity: For classification, measures how often a randomly chosen element would be incorrectly labeled. $G = 1 - \sum_{k=1}^{C} p_k^2$
- Entropy: Measures impurity or disorder. $H = -\sum_{k=1}^{C} p_k \log_2(p_k)$
- Information Gain: Reduction in entropy after splitting on an attribute.
Pruning: Techniques to reduce the size of decision trees to prevent overfitting.

29:["$","main",null,{"className":"pt-[57px]","children":["$","div",null,{"className":"max-w-7xl mx-auto px-4 sm:px-6 py-6","children":[["$","nav",null,{"aria-label":"Breadcrumb","className":"mb-4","children":["$","ol",null,{"className":"flex items-center gap-1 text-sm text-muted-foreground","child