Unified Vision-Based Motion Estimation and Control for Multiple and Complex Robots

Overview

The project considers teams of robots that need to collaborate on a physical task. As a driving application, we consider a Robot Construction Crew that needs to assemble a building using prefabricated components. Each robot might have non-trivial kinematics (e.g., a robotic arm mounted on a non-holonomic platform), and is generally equipped with vision sensors (cameras). To achieve their mission, the robots need to:

  1. Localize themselves with respect to each other, to the construction area, and to the prefabricated components that need to be assembled.
  2. Coordinate their motions to collaboratively grasp, transport and assemble the components
  3. Plan how to survey and inspect the results of their work.

The functions above imply vision-kinodynamic constraints deriving from the intrinsic geometric properties of how the robots move and sense. The main insight of this work is that the constraints above are bilinear in the variables parametrizing the kinematic configurations of the robots and the objects, i.e., in the translations and rotation matrices.

We introduce novel optimization formulations that take advantage of this structure by combining the vision-kinodynamic constraints into convex constraints (e.g., defining a Semi-Definite Program) and non-convex low-rank constraints (from the bilinearity over translations and rotations). We expect that this methodology will surpass current state- of-the-art solutions, by providing robust joint solvers instead of cumbersome and brittle pipelines.

Funding and support

NSF LogoThis project is supported by the National Science Foundation grant “Unified Vision-Based Motion Estimation and Control for Multiple and Complex Robots” (Award number 2212051)

Start date: August 1, 2022
End date: July 31, 2025

Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.