知の物理学研究センター / Institute for Physics of Intelligence (iπ)

【日時/Date】
2023年12月5日（火）10時30分～12時 / Dec. 5, 2023, 10:30 - 12:00 (JST)

【発表者/Speaker】
太田 敏博 氏（ サイバーエージェント AI Lab）

【タイトル/Title】
“Hopfield/Mixer correspondence: towards a better understanding of MetaFormers architecture design”

【概要/Abstract】
In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models (MetaFormers) that compete with Transformers, such as the MLP-Mixer. Despite their weak induced bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this talk, we discuss how the modern Hopfield networks may be useful for a unified understanding of MetaFormer. In particular, we propose Hopfield/Mixer correspondence as a new direction for MetaFormers architecture design, and introduce a novel MetaFormer model theoretically derived from the correspondence. Finally, possible extensions and their prospects will be discussed.