EAGER: Robust Data-Driven Robotic Manipulation via Bayesian Inference and Passivity-Based Control

Robots usually move objects by firmly holding on to them. Some tasks cannot be done this way, because the object may be delicate, or large relative to the robot's arm or hand. For example, we use firm holds when moving a closed book, but delicate finger motions when turning a page. Since the robot's "hand" may move relative to the object, the contact type between the robot, object, and environment can change during manipulation. Forces applied on the object create different motions when contact conditions are different. Conversely, different motions may lead to different contacts in the future. Planning computational methods can identify the right sequence of forces and contact conditions that could complete a task. Small errors that crop up during execution of planned motions would normally be reduced by taking corrective actions. However, these corrective actions often do not account for changing = contacts, and the errors instead are more critical due to unanticipated contacts, ultimately leading to failure on tasks. This EArly-concept Grant for Exploratory Research (EAGER) project will study techniques to create plans for robot motion that mitigate instead of amplify errors during execution of such tasks. Such manipulation tasks involving significant contact events can be found in robotic applications such as loading dishwashers, fetching hard-to-reach objects from cluttered cupboards, or moving furniture. The project team will study new data-driven methods to train robust motion controllers that are derived from Bayesian neural networks with special structure informed by robotics and control principles. To account for the contact-rich nature of the task, the network will consist of a mixture-of-experts, where each expert is either a controller or a storage function used to derive a passivity-based controller. A gating network chooses which controller to use given the input to the network. Bayesian networks will provide a distribution over motor commands given an input, allowing the motion controller to account for uncertainty. The project will proceed in three overlapping stages: The investigators will use tools from formal verification to synthesize controllers that provably locally stabilize contact-rich motion plans, and use these controllers to initialize a prior distribution for the weights of the Bayesian neural network using knowledge distillation. This initialized network will be trained from task-based rewards in an end-to-end manner using data from differentiable simulators, where the robot-object-environment system parameters are uncertain. The trained network will be tested in experiments involving a robot arm pushing a large box over step-like obstacles designed to require changes in contact conditions during manipulation. The project, if successful, will identify a controller synthesis paradigm that simultaneously overcomes the simulation-to-reality gap and the data-inefficiency plaguing purely data-driven approaches for contact-rich object manipulation. This project will also advance knowledge in scaling up computational controller synthesis, and contribute new tools for GPU-accelerated simulation of stochastic systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.