MLT: Machine Learning Techniques

Topics covered

• Introduction to Machine Learning

  • What is machine learning?
  • Types of machine learning problems
  • Supervised and unsupervised learning
  • Applications of machine learning

• Representation Learning

  • Feature representation
  • Data representation in high-dimensional spaces
  • Dimensionality reduction

• Principal Component Analysis (PCA)

  • Motivation and intuition behind PCA
  • Variance and directions of maximum variance
  • Covariance matrix
  • Eigenvalues and eigenvectors
  • Principal components
  • Projection of data onto principal components
  • Dimensionality reduction using PCA
  • Reconstruction and explained variance

Expected skills: Understand the purpose of PCA, identify principal components and use eigenvectors and eigenvalues to reduce the dimensionality of data.

Topics covered

• Review of PCA

  • Linear dimensionality reduction
  • Limitations of ordinary PCA

• Kernel Methods

  • Motivation for kernel methods
  • Nonlinear feature mappings
  • Feature spaces
  • Kernel functions

• Kernel PCA

  • Kernel trick
  • Kernel matrix
  • Centering the kernel matrix
  • Eigenvalue decomposition
  • Nonlinear dimensionality reduction

• Common Kernels

  • Polynomial kernel
  • Gaussian or RBF kernel
  • Kernel parameter selection

• Comparing PCA and Kernel PCA

  • Linear versus nonlinear projections
  • When Kernel PCA is useful
  • Limitations of Kernel PCA

Expected skills: Understand kernel functions, calculate kernel values and explain how Kernel PCA captures nonlinear patterns.

Topics covered

• Introduction to Clustering

  • What is clustering?
  • Similarity and distance measures
  • Clustering versus classification

• K-means Clustering

  • Objective function
  • Centroids and cluster assignments
  • Initialization
  • Assignment and update steps
  • Convergence
  • Choosing the number of clusters
  • Limitations of K-means

• Kernel K-means

  • Motivation
  • Nonlinear cluster boundaries
  • Kernel-based clustering
  • Comparing K-means and Kernel K-means

Expected skills: Perform K-means iterations, calculate cluster centroids, evaluate clustering objectives and understand Kernel K-means.

Topics covered

• Review of Maximum Likelihood Estimation (MLE)

  • Likelihood functions
  • Log-likelihood
  • Parameter estimation

• Bayesian Estimation

  • Prior and posterior distributions
  • Bayesian parameter estimation
  • Difference between MLE and Bayesian estimation

• Gaussian Mixture Models (GMM)

  • Mixture distributions
  • Gaussian components
  • Mixing coefficients
  • Latent variables
  • Responsibilities

• Expectation-Maximization (EM) Algorithm

  • Motivation and intuition
  • E-step
  • M-step
  • Parameter updates
  • Iterative optimization
  • Convergence and limitations

• GMM-based Clustering

  • Soft cluster assignments
  • Comparing GMM and K-means

Expected skills: Understand probabilistic clustering, estimate mixture model parameters and perform the E-step and M-step of the EM algorithm.

Topics covered

• Introduction to Regression

  • Regression problems
  • Input features and target variables
  • Linear regression model
  • Prediction using regression

• Least Squares Regression

  • Linear regression equations
  • Sum of squared errors
  • Least squares objective function
  • Normal equations
  • Matrix formulation
  • Geometric interpretation of least squares

• Bayesian View of Regression

  • Probabilistic interpretation of regression
  • Gaussian noise assumptions
  • Likelihood function
  • Parameter estimation
  • Relationship between least squares and maximum likelihood

• Model Evaluation

  • Residuals
  • Training error
  • Prediction error

Expected skills: Derive and solve least squares regression problems and understand the probabilistic interpretation of regression.

Topics covered

• Limitations of Ordinary Least Squares

  • Overfitting
  • High-dimensional features
  • Multicollinearity
  • Model complexity

• Regularization

  • Motivation for regularization
  • Bias-variance trade-off
  • Regularized objective functions

• Ridge Regression

  • L2 regularization
  • Ridge objective function
  • Closed-form solution
  • Effect of the regularization parameter

• LASSO Regression

  • L1 regularization
  • LASSO objective function
  • Sparse solutions
  • Feature selection

• Comparing Ridge and LASSO

  • Differences between L1 and L2 penalties
  • Coefficient shrinkage
  • Model interpretability

Expected skills: Formulate regularized regression objectives, compare Ridge and LASSO, and understand how regularization affects model coefficients.

Topics covered

• Introduction to Classification

  • Classification problems
  • Class labels
  • Decision boundaries
  • Binary and multiclass classification

• K-Nearest Neighbours (K-NN)

  • Distance-based classification
  • Euclidean distance
  • Nearest neighbours
  • Choosing k
  • Classification using majority voting
  • Effect of distance metrics

• Decision Trees

  • Tree-based classification
  • Decision nodes and leaf nodes
  • Splitting criteria
  • Information gain
  • Entropy
  • Gini impurity
  • Tree construction
  • Overfitting in decision trees

Expected skills: Solve K-NN classification problems, calculate impurity measures and understand how decision trees partition feature space.

Topics covered

• Generative and Discriminative Models

  • Generative modelling
  • Discriminative modelling
  • Joint and conditional probability distributions
  • Class-conditional distributions

• Maximum Likelihood Classification

  • Likelihood-based classification
  • Estimating model parameters
  • Bayes decision rule

• Naive Bayes

  • Conditional independence assumption
  • Naive Bayes classifier
  • Gaussian Naive Bayes
  • Multinomial Naive Bayes
  • Bernoulli Naive Bayes

• Exponential Family

  • Exponential-family distributions
  • Connection with probabilistic modelling
  • Examples of common distributions

• Classification Decisions

  • Posterior probabilities
  • Decision boundaries
  • Classification under different assumptions

Expected skills: Calculate posterior probabilities using Bayes' theorem, apply Naive Bayes and distinguish generative from discriminative models.

Topics covered

• Discriminative Classification

  • Conditional probability models
  • Decision boundaries
  • Linear classifiers

• Perceptron

  • Perceptron model
  • Linear decision function
  • Perceptron learning rule
  • Weight updates
  • Perceptron convergence for linearly separable data
  • Limitations

• Logistic Regression

  • Sigmoid function
  • Logistic regression model
  • Probability interpretation
  • Log-odds
  • Decision boundary
  • Binary classification

• Optimization of Logistic Regression

  • Likelihood
  • Log-likelihood
  • Cross-entropy loss
  • Gradient-based optimization

• Comparing Perceptron and Logistic Regression

  • Differences in objective functions
  • Probabilistic outputs
  • Classification behaviour

Expected skills: Perform perceptron updates, calculate logistic regression probabilities and understand the loss function used for classification.

Topics covered

• Introduction to SVM

  • Maximum-margin classification
  • Separating hyperplanes
  • Support vectors

• Linear SVM

  • Hard-margin SVM
  • Soft-margin SVM
  • Margin maximization
  • Hinge loss
  • Regularization parameter C

• Optimization Formulation

  • Primal formulation
  • Constraints
  • Lagrangian formulation
  • Dual formulation

• Kernel SVM

  • Motivation for kernel methods
  • Kernel trick
  • Nonlinear decision boundaries
  • Polynomial kernels
  • Gaussian or RBF kernels

• SVM Classification

  • Prediction using support vectors
  • Effect of the regularization parameter
  • Model limitations

Expected skills: Calculate margins, identify support vectors and understand the optimization formulation of linear and kernel SVMs.

Topics covered

• Introduction to Ensemble Learning

  • Combining multiple models
  • Ensemble predictions
  • Motivation for ensembles

• Bagging

  • Bootstrap sampling
  • Training multiple models
  • Aggregating predictions
  • Variance reduction

• Boosting

  • Sequential learning
  • Combining weak learners
  • Differences between bagging and boosting

• AdaBoost

  • Weak classifiers
  • Sample weights
  • Weighted classification error
  • Updating sample weights
  • Classifier weights
  • Final weighted prediction

• Comparing Ensemble Methods

  • Bias and variance
  • Model stability
  • Strengths and limitations

Expected skills: Understand bootstrap aggregation, perform AdaBoost weight updates and explain how ensemble learning combines multiple classifiers.

Topics covered

• Introduction to Artificial Neural Networks

  • Artificial neurons
  • Weights and biases
  • Neural network architecture
  • Input, hidden and output layers

• Neural Network Computation

  • Forward propagation
  • Weighted sums
  • Activation functions
  • Nonlinear transformations

• Multiclass Classification

  • Multiclass outputs
  • Softmax function
  • Class probabilities
  • Multiclass prediction

• Loss Functions

  • Cross-entropy loss
  • Multiclass classification loss
  • Objective function

• Neural Network Training

  • Gradient-based learning
  • Backpropagation fundamentals
  • Weight updates
  • Training challenges

• Neural Network Evaluation

  • Classification accuracy
  • Confusion matrix
  • Generalization
  • Overfitting and model complexity

Expected skills: Understand neural network computations, calculate softmax probabilities and explain how neural networks perform multiclass classification.