Adaptive and robust optimisation algorithms for large scale machine learning

In the era of large-scale artificial intelligence (AI) and machine learning (ML), core challenges arise in the optimisation process in terms of scalability, efficiency and reliability. An overarching goal in large scale training is to design fast, robust and adaptive algorithms which are self-adjusting to unknown properties of the model and the dataset, as well as variations in the loss landscape.

In this talk, I will investigate shortcomings of classical approaches and explain how data-driven, adaptive mechanisms could help in theory and application. First, I will present an adaptive stochastic gradient descent method that could automatically adjust its convergence rate with respect to the curvature of the loss function while simultaneously adapting to the unknown noise levels in the gradients. Second, we will consider the more general non-convex setting and I will explain an adaptive mechanism for controlling the noise in gradient computation for faster optimization. Finally, I will talk about a simple and resource-efficient adaptive framework for solving min-max problems, involving multi-agent scenarios, which outperforms existing algorithms in runtime.

Click here to join the seminar via Zoom
Meeting ID: 954 4713 9621
Passcode: 539506

Ali Kavis is a postdoctoral fellow at the University of Texas at Austin in the Department of Electrical and Computer Engineering, working with Sujay Sanghavi and Aryan Mokhtari. He obtained his PhD in Computer and Communication Sciences from École Polytechnique Fédérale de Lausanne (EPFL) in 2023, advised by Volkan Cevher. 

His research targets the algorithmic foundations of ML, and studies theoretical and practical behaviour of adaptive optimisation methods for convex and non-convex minimisation problems as well as min-max problems. He develops robust and efficient algorithms which automatically adapt to the loss landscape, noisy computations and the unknown nature of the data by means of monitoring trajectory-related information on-the-fly.

His work received the Spotlight award four times in ICML and NeurIPS. He is also the recipient of multiple Swiss National Science Foundation (SNSF) individual research grants; Postdoc.Mobility grant (CHF 120,000) and Postdoc.Mobility Return grant (CHF 115,000).

For more information about the ESD Seminar, please email esd_invite@sutd.edu.sg