MLF: Machine Learning Foundations

Topics covered

• Introduction to Machine Learning

  • What is machine learning?
  • Applications of machine learning
  • Data, models and machine learning tasks

• Supervised Learning

  • Regression
  • Classification
  • Training data and prediction

• Unsupervised Learning

  • Dimensionality reduction
  • Density estimation
  • Learning patterns without labelled data

• Machine Learning Problem Formulation

  • Identifying machine learning problems
  • When to use machine learning
  • When not to use machine learning

• Real-world Applications

  • Working with a real-world dataset
  • Dimensionality reduction applications
  • Density estimation applications

Expected skills: Identify machine learning problems, distinguish supervised and unsupervised learning, and understand how data and models are connected.

Topics covered

• Sets and Functions

  • Set operations and functions
  • Domain and range
  • Function composition

• Univariate Calculus

  • Limits
  • Continuity
  • Differentiability
  • Derivatives
  • Geometric interpretation of derivatives

• Derivatives and Linear Approximation

  • Tangent lines
  • Linear approximations
  • First-order approximations

• Applications and Advanced Rules

  • Chain rule
  • Product and quotient rules
  • Higher-order derivatives
  • Applications of derivatives

• Multivariate Calculus

  • Lines and planes in higher-dimensional space
  • Functions of multiple variables
  • Partial derivatives
  • Linear approximation in multiple dimensions
  • Applications of multivariable calculus

Expected skills: Compute derivatives and partial derivatives, understand linear approximations and use calculus in optimization problems.

Topics covered

• Four Fundamental Subspaces

  • Column space
  • Row space
  • Null space
  • Left null space

• Orthogonal Vectors and Subspaces

  • Dot products
  • Orthogonality
  • Orthogonal complements

• Projections

  • Projection of a vector onto a subspace
  • Projection matrices
  • Orthogonal projections

• Least Squares

  • Linear systems with no exact solution
  • Least squares approximation
  • Normal equations
  • Projection onto column spaces

• Solving Linear Systems

  • Reduced row echelon form
  • Row space computations
  • Solutions to Ax=b

• Connection with Regression

  • Linear regression in matrix form
  • Least squares as a regression objective
  • Geometric interpretation of regression

Expected skills: Solve least squares problems, compute projections and connect linear algebra to regression.

Topics covered

• Eigenvalues and Eigenvectors

  • Definition of eigenvalues
  • Definition of eigenvectors
  • Characteristic equation
  • Characteristic polynomial

• Eigenvalue Computation

  • Determinants and eigenvalues
  • Finding eigenvalues of 2×2 matrices
  • Finding corresponding eigenvectors

• Diagonalization

  • Diagonalizable matrices
  • Similarity transformations
  • Eigenvector matrices
  • Diagonal matrix representation

• Applications of Diagonalization

  • Matrix powers
  • Fibonacci sequence using diagonalization
  • Understanding repeated matrix operations

• Orthogonal Diagonalization

  • Conditions for diagonalization
  • Orthogonal eigenvectors
  • Geometric interpretation

Expected skills: Calculate eigenvalues and eigenvectors, diagonalize suitable matrices and apply diagonalization to matrix powers.

Topics covered

• Complex Matrices

  • Complex-valued matrix entries
  • Conjugate transpose
  • Properties of complex matrices

• Hermitian Matrices

  • Definition of Hermitian matrices
  • Properties of Hermitian matrices
  • Eigenvalues of Hermitian matrices

• Unitary Matrices

  • Definition and properties
  • Unitary transformations
  • Relationship with orthogonality

• Diagonalization of Hermitian Matrices

  • Spectral decomposition
  • Orthogonal or unitary eigenvector bases
  • Diagonalization of Hermitian matrices

• Special Matrix Properties

  • Real symmetric matrices
  • Orthogonal diagonalization
  • Applications of symmetric matrices

Expected skills: Identify symmetric and Hermitian matrices, understand their eigenvalue properties and apply spectral decomposition.

Topics covered

• Singular Value Decomposition (SVD)

  • Definition of SVD
  • Singular values
  • Left and right singular vectors
  • Matrix factorization

• Computing SVD

  • Relationship between SVD and eigenvalue decomposition
  • Singular values using A^TA and AA^T
  • SVD of small matrices

• Geometric Interpretation of SVD

  • Rotations and scaling
  • Transformation of geometric shapes
  • Low-rank approximation

• Positive Definiteness

  • Positive definite matrices
  • Positive semidefinite matrices
  • Eigenvalue criteria
  • Quadratic forms

• PCA in Image Processing

  • Image representation
  • Dimensionality reduction
  • Image compression
  • Reconstruction using principal components

• Positive Definite Functions and Matrices

  • Properties and examples
  • Connections with matrix factorization

Expected skills: Compute SVD for small matrices, understand low-rank approximations and connect PCA with image compression.

Topics covered

• Introduction to Optimization

  • Optimization problems
  • Objective functions
  • Parameters and decision variables
  • Local and global optima

• Unconstrained Optimization

  • Minimization and maximization
  • First-order optimality conditions
  • Second-order conditions

• Gradient Descent

  • Gradient of a function
  • Gradient descent algorithm
  • Learning rate
  • Iterative parameter updates

• Gradient Descent for Multivariable Functions

  • Partial derivatives
  • Gradient vectors
  • Multivariable gradient descent

• Taylor Series

  • First-order Taylor approximation
  • Second-order Taylor approximation
  • Taylor series in higher dimensions
  • Connection between Taylor approximations and gradient descent

• Failure Modes

  • Overshooting
  • Slow convergence
  • Local minima and saddle points
  • Effect of learning rate

Expected skills: Formulate unconstrained optimization problems, compute gradients, perform gradient descent iterations and recognize common optimization failure modes.

Topics covered

• Introduction to Convexity

  • Convex sets
  • Convex combinations
  • Line segments between points
  • Convex and non-convex sets

• Properties of Convex Sets

  • Intersections of convex sets
  • Affine sets
  • Convex hulls
  • Geometric interpretation

• Convex Functions

  • Definition of convex functions
  • Strictly convex functions
  • Concave functions
  • First-order characterization
  • Second-order characterization

• Properties of Convex Functions

  • Local minima and global minima
  • Convexity and optimization
  • Examples of convex and non-convex functions

• Convex Optimization Problems

  • Objective functions and feasible regions
  • Convex constraints
  • Convex optimization formulation
  • Applications in machine learning

Expected skills: Test convexity, identify convex sets and functions, and understand why convexity matters in optimization.

Topics covered

• Constrained Optimization

  • Equality constraints
  • Inequality constraints
  • Feasible regions
  • Constrained objective functions

• Lagrange Multipliers

  • Lagrangian function
  • Lagrange multiplier conditions
  • Solving equality-constrained optimization problems
  • Geometric interpretation

• Projected Gradient Descent

  • Constrained gradient updates
  • Projection onto feasible sets
  • Iterative optimization

• Logistic Regression as an Optimization Problem

  • Logistic regression objective
  • Log-likelihood
  • Cross-entropy loss
  • Gradient-based optimization

• Applications

  • Optimization under constraints
  • Interpreting optimality conditions

Expected skills: Solve basic constrained optimization problems using Lagrange multipliers and understand logistic regression as an optimization problem.

Topics covered

• Introduction to Probabilistic Models

  • Probability models in machine learning
  • Random variables and distributions
  • Probability density functions

• Continuous Random Variables

  • Continuous probability distributions
  • Conditional PDFs
  • Conditional probability density

• Expectation

  • Expected values
  • Expectation of functions
  • Properties of expectation

• Multiple Random Variables

  • Joint distributions
  • Marginal distributions
  • Independent random variables
  • Conditional distributions

• Transformed Random Variables

  • Functions of random variables
  • Transformations of continuous variables
  • Distribution of transformed variables

• Common Probability Distributions

  • Uniform distribution
  • Exponential distribution
  • Normal distribution

• Probabilistic Models in ML

  • Modelling data using distributions
  • Conditional models
  • Examples of probabilistic learning

Expected skills: Understand probabilistic modelling, calculate basic expectations and interpret distributions used in machine learning.

Topics covered

• Introduction to the Exponential Family

  • General exponential-family form
  • Natural parameters
  • Sufficient statistics
  • Log-partition function

• Exponential-Family Distributions

  • Bernoulli distribution
  • Binomial distribution
  • Gaussian distribution
  • Poisson distribution

• Properties of the Exponential Family

  • Mean parameters
  • Variance relationships
  • Moments
  • Sufficient statistics

• Applications in Machine Learning

  • Probabilistic classification
  • Generalized linear models
  • Exponential-family likelihoods

• Understanding Distribution Parameters

  • Natural and standard parameters
  • Parameter interpretation
  • Relationships between distributions

Expected skills: Express common probability distributions in exponential-family form and understand their important statistical properties.

Topics covered

• Parameter Estimation

  • Statistical models
  • Unknown parameters
  • Likelihood functions
  • Maximum Likelihood Estimation
  • Estimating model parameters from observed data

• Gaussian Mixture Models

  • Mixture distributions
  • Gaussian components
  • Mixing coefficients
  • Latent variables

• Expectation-Maximization (EM)

  • Motivation for EM
  • Latent-variable models
  • E-step
  • M-step
  • Iterative parameter updates

• EM for Gaussian Mixture Models

  • Responsibilities
  • Estimating means
  • Estimating covariance matrices
  • Estimating mixture weights

• Laws of Large Numbers and Probability Bounds

  • Markov's inequality
  • Chebyshev's inequality
  • Hoeffding's inequality
  • Law of Large Numbers
  • Central Limit Theorem

• Solved Examples

  • Parameter estimation exercises
  • Gaussian mixture examples
  • EM iterations

Expected skills: Estimate model parameters, understand Gaussian mixture models and perform the main steps of the EM algorithm.