Form2Fit: Learning Shape Priors for Generalizable Assembly

from Disassembly

Blog Paper Video Code Slides

Is it possible to learn policies for robotic assembly that can generalize to new objects? In this work, we propose to formulate the kit assembly task as a shape matching problem, where the goal is to learn a shape descriptor that establishes geometric correspondences between object surfaces and their target placement locations from visual input. This formulation enables the model to acquire a broader understanding of how shapes and surfaces fit together for assembly -- allowing it to generalize to new objects and kits. To obtain training data for our model, we present a self-supervised data-collection pipeline that obtains ground truth object-to-placement correspondences by disassembling complete kits. Our resulting real-world system, Form2Fit, learns effective pick and place strategies for assembling objects into a variety of kits -- achieving 90% average success rates under different initial conditions (e.g. varying object and kit poses), 94% success under new configurations of multiple kits, and over 86% success with completely new objects and kits.

Highlights

Paper

★ Best Paper Award in Automation Finalist ★

To appear at the IEEE International Conference on Robotics and Automation (ICRA) 2020.

Latest version (Oct. 31, 2019): arXiv:1910.13675 [cs.RO] or here.

You can view the supplemental material here.

Team

Kevin Zakka 1,2
Andy Zeng 2, 3
Johnny Lee 2
Shuran Song 2,3
1 Stanford University            2 Google            3 Columbia University            

Code and Extras

You can find additional resources on Github. Includes:

  • The Form2Fit Benchmark
  • Paper code (architectures, losses, dataloaders, etc.)
  • Model weights

Bibtex

@inproceedings{zakka2020form2fit,
  title={Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly},
  author={Zakka, Kevin and Zeng, Andy and Lee, Johnny and Song, Shuran},
  booktitle={Proceedings of the IEEE International Conference on Robotics and Automation},
  year={2020}
}

Technical Summary Video (with audio)

Example Results: Varying Initial Conditions

Trained only on single kit poses (i.e. position and orientation), Form2Fit can successfully assemble kits with random object and kit poses.

Single-Object Kit
Single-Object Kit
Order Matters
Order Doesn't Matter

Example Results: Generalizing to Novel Configurations

A model trained on individual kits can learn to assemble combinations and mixtures of these kits.

Example Results: Generalizing to Novel Objects & Kits

Form2Fit can also assemble previously unseen objects and kits.

Never-Before-Seen Zoo Animals

Acknowledgements

We would like to give special thanks to Nick Hynes, Alex Nichol, and Ivan Krasin for fruitful technical discussions, Adrian Wong, Brandon Hurd, Julian Salazar, and Sean Snyder for hardware support, and Ryan Hickman for valuable managerial support. We are also grateful for hardware and financial support from Google.

Contact

If you have any questions, please feel free to contact Kevin Zakka at zakka@cs.stanford.edu.

Last Edit: Feb. 16, 2020.