知の物理学研究センター / Institute for Physics of Intelligence (ipi)

【Date】
Dec. 5, 2023 10:30 - 12:00 (JST)
via zoom & #415, #1 Faculty of Science Building

【Speaker】
Toshihiro Ota, Cyber Agent AI Lab

【Title】
“Hopfield/Mixer correspondence: towards a better understanding of MetaFormers architecture design”

【Abstract】
In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models (MetaFormers) that compete with Transformers, such as the MLP-Mixer. Despite their weak induced bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this talk, we discuss how the modern Hopfield networks may be useful for a unified understanding of MetaFormer. In particular, we propose Hopfield/Mixer correspondence as a new direction for MetaFormers architecture design, and introduce a novel MetaFormer model theoretically derived from the correspondence. Finally, possible extensions and their prospects will be discussed.

*To receive the Zoom invitation and monthly reminders, please register via this google form: https://forms.gle/dqxhpsZXLNYvbSB38

Your e-mail addresses will be used for this purpose only, you can unsubscribe anytime, and we will not send more than three e-mails per month.

Takashi Takahashi and Ken Nakanishi

⇦Top page