Data Validation & ML Cheatsheets

Cheatsheet Content

### Data Validation in Statistics Data validation ensures data quality and reliability for statistical analysis. #### 1. Data Cleaning - **Missing Values:** - **Imputation:** Mean, median, mode, regression, K-NN. - **Deletion:** Row-wise, column-wise (if too many missing). - **Outliers:** - **Detection:** Z-score, IQR method, box plots, scatter plots. - **Treatment:** Capping, flooring, transformation, removal (with caution). - **Inconsistent Data:** - **Standardization:** Units, formats (e.g., date formats). - **Normalization:** Scaling features to a standard range (e.g., Min-Max, Z-score). #### 2. Data Integrity Checks - **Uniqueness:** Identify and remove duplicate records. - **Validity:** Ensure data conforms to defined rules (e.g., age > 0, score ### Predictive Analysis & Forecasting Using historical data to make predictions about future outcomes. #### 1. Key Concepts - **Target Variable:** The variable to be predicted. - **Features (Predictors):** Variables used to predict the target. - **Time Series Data:** Data collected over a period of time. - **Cross-Sectional Data:** Data collected at a single point in time. #### 2. Common Techniques - **Regression Analysis:** - **Linear Regression:** $Y = \beta_0 + \beta_1 X_1 + ... + \epsilon$ - **Multiple Regression:** Extends linear regression to multiple predictors. - **Logistic Regression:** For binary classification (predicting probability). - **Time Series Models:** - **ARIMA (AutoRegressive Integrated Moving Average):** - p: order of the AutoRegressive part - d: degree of differencing (I for Integrated) - q: order of the Moving Average part - **SARIMA:** Seasonal ARIMA for data with seasonal patterns. - **Exponential Smoothing:** Holt-Winters for trend and seasonality. - **Decision Trees/Random Forests:** For both regression and classification. - **Neural Networks:** Deep learning models for complex patterns. #### 3. Evaluation Metrics - **Regression:** MAE, MSE, RMSE, $R^2$. - **Classification:** Accuracy, Precision, Recall, F1-Score, AUC-ROC. ### Machine Learning Algorithms Build Techniques Steps to build and deploy ML models. #### 1. Problem Definition & Data Collection - Clearly define the objective (classification, regression, clustering). - Gather relevant data from various sources. #### 2. Data Preprocessing - **Data Cleaning:** Handle missing values, outliers, inconsistencies (as above). - **Feature Engineering:** Create new features, select important ones. - **Data Scaling:** Normalization (Min-Max) or Standardization (Z-score). - **Data Splitting:** Train-validation-test sets (e.g., 70/15/15 or 80/20). #### 3. Model Selection & Training - **Algorithm Choice:** Based on problem type and data characteristics (e.g., Logistic Regression for binary classification, SVM, K-NN, Gradient Boosting). - **Training:** Fit the chosen model to the training data. - **Cross-Validation:** K-fold cross-validation to assess model performance robustly. #### 4. Model Evaluation - Use appropriate metrics on the test set (see Predictive Analysis section). - **Confusion Matrix:** For classification problems. - **Bias-Variance Trade-off:** - **High Bias (Underfitting):** Model too simple, can't capture underlying patterns. - **High Variance (Overfitting):** Model too complex, learns noise in training data. #### 5. Hyperparameter Tuning - **Grid Search:** Exhaustively searches a specified parameter space. - **Random Search:** Randomly samples parameters from a distribution. - **Bayesian Optimization:** Uses probability to find optimal hyperparameters. #### 6. Model Deployment & Monitoring - Deploy the trained model into a production environment. - Continuously monitor performance and retrain as needed.