Beyond Scaling Up: Exploration, Guidance, and Symmetry in Robot Perception

Abstract

Recent generalist robot systems rely on vision-language-action models without making any use of perception capabilities like 3D or 4D representations encoded in vision foundation models. They increasingly rely on scaling up the number of examples needed for behavior cloning, not only to capture the distribution of tasks but also basic perceptual skills. We argue that a robot should be an active observer that selects the best views required for scene representation and the affordances involved in the task at hand. Such an exploration can rely on information-theoretic principles that guide the robot towards unpredictable views. Moreover, test-time geometric reasoning can adapt to arbitrary environments, enabling collision-free planning and one-shot adaptation. Last, symmetry enables better generalization and learning dynamics. We propose an equivariant anonicalization framework with applications in trajectory planning and odometry.