Robotic Manipulation of Deformable Objects

An introduction



Robotic manipulators, also known as industrial robots, are of wide use in today’s industry. However, due to their precise nature, it has been difficult for industrial robots to manipulate. This work is conducted at MSC Lab of UC Berkeley, and sponsored by FANUC corporation.

Work Overview

In our work, we teach robots how to manipulate a certain object step by step. Then we change the scene, with a similar object but with slightly different shape, size or direction. The robot is able to transfer what it learns by human demonstration to a desirable trajectory that can manipulate the new object to a target state.

The major challenges are

  • object perception
  • trajectory generation
  • logic inference

Basically, we are able to get the point cloud information gathererd by Microsoft Kinect. Then we need to deduct the position of our object. Sometimes our object is partly occluded (which most always happens, as our robot arm is manipulating it). We need to still get its shape and position under these circumstances. Computer vision libraries alone are not enough to solve these problems and we needed to develop new algorithms. For this work, we used CPD algorithm and bullet physics engine to power our perception. See refrence for more detailed information.

For robot manipulation, we applied different methods. But the basic idea is to warp the training trajectory to test scenes. The warping function is derived from the object states of training scene and test scene. The features of these states are extracted to help the robot move in a way that can preserve certain features of the manipulated object. For example, mapping the tangent space (instead of cartesian space) feature of a rope will help preserve its length, avoiding the accidents of over-stretching.

These will get clearer in other articles describing specific work.


With these algorithms implemented, we are able to let two robots collaborate and

  • manipulate a rope to a desired shape
  • knot a rope
  • fold clothes
  • many more… videos can be found both on this website and on MSC Lab’s site

Code and Distribution Rules

Part of the codes for this project can be found at my github. Codes are written in MATLAB. Note that due to copyright issues, you may not publish or spread related information (videos, codes, etc.) without our permission. Visit MSC Lab‘s website for detailed information, raw materials, and contact method.

Working together, we have accomplished far beyond what has been stated here on my website (which is merely my portion of work). Below is a photo of some members of our great team!

FANUC Family 2017 Summer


Some of the papers are not published yet. (as of August, 2017) Published papers related to this work and briliant work from former researchers include:

[1] T. Tang, C. Liu, W. Chen and M. Tomizuka, “Robotic manipulation of deformable objects by tangent space mapping and non-rigid registration,” 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, 2016, pp. 2689-2696.

[2] A. Myronenko and X. Song, “Point Set Registration: Coherent Point Drift,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 12, pp. 2262-2275, Dec. 2010.
doi: 10.1109/TPAMI.2010.46

[3] Haili Chui, Anand Rangarajan, A new point matching algorithm for non-rigid registration, Computer Vision and Image Understanding, Volume 89, Issue 2, 2003, Pages 114-141, ISSN 1077-3142,