Building a robot to function in the real-world remains challenging, which requires (1) perception, that recognizes the hidden states of the embodied scene, (2) decision making, that generates optimal plans to achieve tasks, and (3) action, that controls robot’s machinery to move in and interact with the environment. With decades of efforts, scientists have introduced numerous techniques that enable robots working in a controlled and known environment (e.g. industrial robots). However, these techniques fail in non-controlled and non-structured environments, such as streets/kitchens/homes/offices that are noisy, complex and constantly changing. Moreover, these techniques are incapable of building robots that continually learn new tasks, adapt effectively to new environments, and achieve each user’s individual need.

This course provides a technical overview of perception, decision making and action control in robotics with an emphasis on the development of generalist robots. The course covers: (1) state-of-the-art robot perception that senses the environment with visual, audio, and tactile signals, (2) classical control and planning, that provides a theoretical basis, and (3) state-of-the-art robot learning that facilitate adaptive and continual learning of robot tasks.