知の物理学研究センター / Institute for Physics of Intelligence (ipi)

【Date】
9:00-10:30, Thur. Dec. 8

【Speaker】
Tomer Galanti/Massachusetts Institute of Technology

【Title】
"On the Role of Neural Collapse in Transfer Learning"

【Abstract】
In a variety of machine learning applications, we have access to a limited amount of data from the task that we would like to solve, as labeled data is often scarce and/ or expensive. In such cases, training directly on the available data is unlikely to produce a model that performs well on new, unseen test samples.
A prominent solution to this problem is to apply transfer learning. In transfer learning, we typically pre-train a foundation model on a given large-scale source task (e.g., ImageNet) and fine-tune it to fit the available data from the downstream task. Recent results show that representations learned by a single classifier over many classes can adapt to new classes with very few samples.
In this talk, we provide an explanation for this behavior based on the recently observed phenomenon of neural collapse. We demonstrate both theoretically and empirically that neural collapse generalizes to new samples from the training classes, and - more importantly - to new classes as well, allowing foundation models to provide feature maps that work well in transfer learning and, specifically, in the few-shot setting.
This work is based on the following publications:
1. T. Galanti, A. Gyorgy, M. Hutter. "On the Role of Neural Collapse in Transfer Learning", ICLR 2022.
2. T. Galanti, A. Gyorgy, M. Hutter. "Improved Generalization Bounds for Transfer Learning via Neural Collapse", ICML Workshop on Pre-Training: Perspectives, Pitfalls, and Paths Forward 2022.
3. C. Xu, S. Yang, T. Galanti, B. Wu, X. Yue, B. Zhai, W. Zhan, P. Vajda, K. Keutzer, M. Tomizuka. "Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models", ECCV 2022.

PDF