ABSTRACT: The rapid rise of deep neural networks raises numerous challenges, both for theoreticians and practitioners. In this talk, I will discuss two such challenges through the perspective of optimization. First, learning the parameters of deep neural networks is a challenging task due to, for instance, the non-convex nature of the high-dimensionality of the objective function. In certain tasks, traditional first-order optimization methods are not well-suited, as the gradient signal might simply not provide sufficient information. I will discuss how alternative optimization methods (such as high-order methods or random search techniques) have the potential to address such shortcomings, both theoretically and practically. Another key challenge is the design of deep neural networks which is largely driven by empirical observations while our theoretical understanding of such models is still behind. I will show how optimization techniques can be used to provide insights into the architecture of a neural network. I will discuss how certain techniques (such as Batch normalization and residual connections) can provably impact the optimization landscape of a neural network.

BIO: Aurelien Lucchi is a researcher at the institute of Machine Learning at ETH Zurich since July 2018. He has earned his PhD in Machine Learning and Computer vision from EPFL in 2013, and a MSc in Computer science from INSA Lyon - France. From January 2014 to June 2018, he was a postdoctoral fellow at ETH Zurich in the group of Prof. Thomas Hofmann. His research interests are in optimization and large scale learning as well as machine learning applications in computer vision, cosmology, quantum computing, etc. He also regularly serves as area chair or program committee member at major conferences (NIPS, ICML, IJCAI) and is a reviewer for major machine learning conferences and journals (JMLR, ICML, NIPS, etc). In addition to his academic career, Aurelien has had significant experience in industrial research including internships at Google Research as well as Microsoft Research.