Words into Action: Learning Diverse Humanoid Robot Behaviors using Language Guided Iterative Motion Refinement

K. Niranjan Kumar, Irfan Essa, Sehoon Ha
Georgia Institute of Technology
Paper arXiv


Abstract

We present a method to simplify controller design by enabling users to train and fine-tune robot control policies using natural language commands. We first learn a neural network policy that generates behaviors given a natural language command, such as “walk forward”, by combining Large Language Models (LLMs), motion retargeting, and motion imitation. Based on the synthesized motion, we iteratively fine-tune by updating the text prompt and querying LLMs to find the best checkpoint associated with the closest motion in history.

BibTeX

@misc{kumar2023words,
      title={Words into Action: Learning Diverse Humanoid Robot Behaviors using Language Guided Iterative Motion Refinement}, 
      author={K. Niranjan Kumar and Irfan Essa and Sehoon Ha},
      year={2023},
      eprint={2310.06226},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}