Computer vision evals

Visual SSL 1.0

Linear probing + end-to-end fine-tuning

Robustness: ImageNet-R, ImageNet-A, ImageNet-Sketch, ImageNet-C

Object classification: ImageNet and variants (ReaL, V2), iNaturalist, Places

Object detection: COCO, RefCOCO-M, PASCAL VOC

Object segmentation: COCO, ADE20K (semantic)

Instance segmentation: Cityscapes

Visual SSL 2.0 (multimodal time!)

https://www.youtube.com/watch?v=hnu-mRLebhc

1. Is CLIP sufficient for multimodal models? (11:03 - 17:20) The speaker investigates whether standard CLIP checkpoints are adequate for visual grounding. By creating a benchmark based on ‘CLIP-blind’ image pairs, the lecture demonstrates that these models often struggle with basic spatial and objective questions, highlighting a need for better vision-centric representations.
2. How can we move beyond static global feature extractors? (19:16 - 27:36) The lecture discusses the importance of ‘deliberate’ visual processing. It proposes a framework that incorporates a visual search mechanism—allowing the model to zoom in, crop, and search for specific targets—rather than relying solely on a single-pass global image feature.
3. What are the necessary components of vision-centric multimodal models? (28:11 - 40:37) Through the ‘Cuber’ study, the talk evaluates various visual backbones, connectors, and instruction-tuning recipes. It emphasizes that evaluation protocols must shift from general benchmarks to more granular, vision-centric tasks to truly measure progress in representation learning.
4. Can multimodal models ‘think in space’? (40:43 - 50:41) A major focus is on spatial reasoning and 3D understanding. The research questions whether current multimodal models can remember and recall spaces effectively, noting that they often fail at long-form video reasoning tasks that require a unified ‘world model’ rather than just a series of isolated local observations.
5. What is the synergy between representation learning and generative modeling? (51:00 - 56:22) The lecture explores how representation learning can improve generative models (like Diffusion Transformers) and conversely, how generative objectives might influence representation quality.