40.303 Nonlinear Optimisation
This course provides an in-depth study of nonlinear optimisation, spanning smooth unconstrained optimisation, convex modelling, constrained optimisation, and large-scale methods used in machine learning. Students develop the mathematical foundations and computational skills needed to analyse and implement gradient, momentum, stochastic, preconditioned, and interior-point methods. The course connects classical concepts such as optimality conditions, convexity, duality, and Newton systems with automatic differentiation and modern adaptive optimisers. Emphasis is placed on selecting methods appropriate to problem structure, understanding convergence and computational trade-offs, and applying the methods to data science, engineering, and machine-learning problems.
Learning objectives
- Classify nonlinear optimisation problems and distinguish global, local, and stationary solutions.
- Apply first- and second-order optimality conditions and recognize convex and strongly convex problems.
- Analyse and implement gradient, momentum, stochastic, and preconditioned optimisation methods.
- Formulate constrained convex problems using disciplined convex programming.
- Use Lagrangian duality, KKT conditions, and interior-point methods to analyse and solve constrained problems.
- Use automatic differentiation to compute gradients for computational models.
- Explain the main ingredients of modern machine-learning optimisers, including adaptive scaling and moment estimation.
- Evaluate optimisation algorithms through computational experiments and a substantial course project.
Measurable outcomes
- Implement and compare at least three nonlinear optimisation algorithms, reporting convergence behaviour and computational cost.
- Formulate a constrained convex problem, verify its optimality conditions, and solve it using an appropriate convex optimisation tool.
- Use automatic differentiation and stochastic gradients in a machine-learning optimisation problem.
- Complete and present a course project that connects optimisation theory, implementation, and empirical evaluation.
Prerequisites
- 40.002 Optimisation
- Understanding of analytical concepts such as Taylor expansion, derivatives, alongside linear algebra essentials like eigenvalues and singular value decomposition. Course examples from MIT are provided:
Course instructor
Number of credits: 12