\u3010Date\u301110:30-12:00 JST, Thursday, Oct. 28<\/p>\n \u3010Title\u3011\"Design space of a deep neural network - its spatial evolution and robustness\" In this talk, I discuss how one can extend the statistical mechanics approach pioneered by E. Garder\u00a0for a single perceptron [2] based on the replica method to a multi-layered, deep perceptron network [3].\u00a0This amounts to\u00a0construct\u00a0a theory in which the Parisi order function is allowed to evolve in space.\u00a0Specifically, I discuss two scenarios: (1) random training data (2) teacher-student setting.\u00a0In both cases, we found the magnitude of the order parameter evolves in space like in 'wetting transitions':\u00a0it is larger closer to the input\/output boundaries suggesting that the effect of the constraints put by the\u00a0training data\u00a0are\u00a0stronger there. If the system is deep enough, the central region remains in the liquid phase\u00a0meaning that the design space remains very large there. Furthermore, in scenario (1) we found a peculiar\u00a0replica symmetry breaking (RSB) which evolves in space: the design space is clustered in a complex,\u00a0hierarchical manner around the boundaries which become progressively simplified approaching the center.\u00a0More recently we found the same type of spatially evolving RSB also in scenario (2) in the presence of noise\u00a0in the training data. But the latter RSB disappears if the network is made deep enough so that the liquid phase\u00a0survives in the center. Finally, I discuss the implications of the theoretical results on deep learning in practice.<\/p>\n [1] G. Parisi, Phys. Rev. Lett. 43, 1754 (1979). The crucial work for the Nobel prize of this year.
\u3010Speaker\u3011Hajime YOSHINO (Osaka University\uff09
<\/span><\/p>\n
\u3010Abstract\u3011
Deep neural networks have a large number of free parameters such as synaptic weights.\u00a0Even with fixed architecture, an extremely large number of different machines can be generated by changing the parameters.\u00a0Learning amounts to\u00a0focus\u00a0on a subset of them which meets constraints imposed by a set of training data.\u00a0Increasing the number of the training data, the design space,\u00a0the\u00a0phase space of the valid machines become smaller.\u00a0It may also split into clusters - glass transition - if the frustration effect of the constraints is severe.\u00a0The Parisi order parameter function [1], developed first for spin-glasses, is useful to capture the nature of such a complex phase space.<\/p>\n
\u3000\u2192see\u00a0https:\/\/www.nobelprize.org\/prizes\/physics\/2021\/summary\/<\/a>
[2] E. Gardner, J. Phys. A: Math. Gen. 21, 257 (1988), E. Gardner and B. Derrida, J. Phys. A: Math. Gen. 22, 1983 (1989).
[3] H. Yoshino, SciPostPhysCore 2, 005 (2020).<\/p>\n