【数学所】Embedding Principle of Loss Landscape of Deep Neural Networks

ON2022-05-17文章来源 数学科学研究所CATEGORY活动

TimeFriday, May 20th, 2022, 10:00-11:00

LocationTecent Meeting

Speaker:  Yaoyu ZhangShanghai Jiao Tong University

Link: https://meeting.tencent.com/dm/N9BlY0oqqrMp 

Room Number: 778-1386-4492

Abstract: Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously important. In this talk, I will present the embedding principle that the loss landscape of a DNN contains all the critical points of all the narrower DNNs. Empirically, we find that a wide DNN is often attracted by highly-degenerate critical points that are embedded from narrow DNNs. The embedding principle provides a new perspective to study the general easy optimization of wide DNNs and unravels a potential implicit low-complexity regularization during the training, by which a more exact and comprehensive understanding can be anticipated in the near future.