Empirics-based Line Searches for Deep Learning

DSpace Repository

Show simple item record

dc.contributor.advisor Zell, Andreas (Prof. Dr.)
dc.contributor.author Mutschler, Maximus
dc.date.accessioned 2023-02-28T10:25:26Z
dc.date.available 2023-02-28T10:25:26Z
dc.date.issued 2023-02-28
dc.identifier.uri http://hdl.handle.net/10900/137091
dc.identifier.uri http://nbn-resolving.de/urn:nbn:de:bsz:21-dspace-1370918 de_DE
dc.identifier.uri http://dx.doi.org/10.15496/publikation-78442
dc.description.abstract This dissertation takes an empirically based perspective on optimization in deep learning. It is motivated by the lack of empirical understanding of the loss landscape's properties for typical deep learning tasks and a lack of understanding of why and how optimization approaches work for such tasks. We solidified the empirical understanding of stochastic loss landscapes to bring color to these white areas on the scientific map with empiric observations. Based on these observations, we introduce understandable line search approaches that compete with and, in many cases outperform, state-of-the-art line search approaches introduced for the deep learning field. This work includes a comprehensive introduction to optimization focusing on line searches in the deep learning field. Based on and guided by this introduction, empirical observations of typical image-classification benchmark tasks' loss landscapes are presented. Further, observations of how optimizers perform and move on such loss landscapes are given. From these observations, the line search approaches Parabolic Approximation Line Search (PAL) and Large Batch Parabolic Approximation Line Search (LABPAL) are derived. In particular, the latter method outperforms all competing line searches in this field in most cases. Furthermore, these observations reveal that well-tuned Stochastic Gradient Descent is already well approximating an almost exact line search, which in parts explains why it is so hard to beat. Given the empirical observations made, it is straightforward to comprehend why and how our optimization approaches work. This contrasts the methodology of many optimization papers in this field which builds upon non-empirically justified theoretical assumptions. Consequently, a general contribution of this work is that it justifies and demonstrates the importance of empirical work in this rather theoretical field. en
dc.language.iso en de_DE
dc.publisher Universität Tübingen de_DE
dc.rights ubt-podok de_DE
dc.rights.uri http://tobias-lib.uni-tuebingen.de/doku/lic_mit_pod.php?la=de de_DE
dc.rights.uri http://tobias-lib.uni-tuebingen.de/doku/lic_mit_pod.php?la=en en
dc.subject.ddc 004 de_DE
dc.subject.other Line Search en
dc.subject.other Deep Learning en
dc.subject.other Stochastic Optimization en
dc.title Empirics-based Line Searches for Deep Learning en
dc.type PhDThesis de_DE
dcterms.dateAccepted 2023-02-16
utue.publikation.fachbereich Informatik de_DE
utue.publikation.fakultaet 7 Mathematisch-Naturwissenschaftliche Fakultät de_DE
utue.publikation.noppn yes de_DE

Dateien:

This item appears in the following Collection(s)

Show simple item record