Abstract
Humanoid agents often struggle to handle flexible and diverse interactions in open environments. A common solution is to collect massive datasets to train a highly capable model, but this approach can be prohibitively expensive. In this paper, we explore an alternative solution: empowering off-the-shelf Vision-Language Models (VLMs, such as GPT-4) to control humanoid agents, thereby leveraging their strong open-world generalization to mitigate the need for extensive data collection. To this end, we present \textbf\{BiBo\} (\textbf\{B\}uilding humano\textbf\{I\}d agent \textbf\{B\}y \textbf\{O\}ff-the-shelf VLMs). It consists of two key components: (1) an \textbf\{embodied instruction compiler\}, which enables the VLM to perceive the environment and precisely translate high-level user instructions (e.g., \{\small\itshape ``have a rest''\}) into low-level primitive commands with control parameters (e.g., \{\small\itshape ``sit casually, location: (1, 2), facing: 90''\}); and (2