Description

In this project, my team and I built a ROS (Robot Operating System) package that allowed a Rethink Baxter robot to sort bottles and cans located in front of it, and drop them into separate recycling bins. We used computer vision to detect and locate several randomly placed objects, and used MoveIt methods to control the robot's kinematics.

My rule was to program the object recognition and classification software and to build the segmentation and calibration library, which was later used by the robot to localize the objects.

Team members: Kailey Smith, Chris Aretakis, Jake Ketchum, Mingqing Yuan, and me.

Take a look at the project on my GitHub page.

The Baxter in action

System Overview

In order to recognize and separate the objects into two different recycle bins, our program is going through a number of steps:

1. Object detection
2. Object segmentation and calibration
3. Robot operation

Step I: Objects Detection

In this step, we used a Realsense D435i camera to get a real-time image of the randomly placed objects. We used OpenCV and image processing libraries to detect and differentiate the objects into 2 groups - cans and bottles.

The following video demonstrates the object differentiation software for several types of cans and bottles:

Recognizing and classifying bottles and cans

Step II: Objects Segmentation and Calibration

In this step, we used the OpenCV library to find the objects segmentation (pixel location in the image). We also located 2 known calibration points (marked in green in the following image) to help us convert objects coordinates from pixels to meters.

The following image demonstrates the differentiation between bottles, cans, and calibrations points:

Recognizing and classifying bottles and cans

Step III: Robot Operation

In this step, we used MoveIt! library for the robot's motion planning and manipulation. The robot moves to the pre-detected objects' locations, grasps one object at a time, and throws it into the appropriate recycle bin. The robot repeats those actions until it finishes picking up all the objects on the table.

The simulated robot in Rviz

The following video demonstrates the robot motion:

Sorting bottles and cans

Future Work

Later, as part of a machine learning based project - Real-Time Object Recognition and Classification, I returned to the Recycler Baxter projects and tried to solve the perception pipeline problems.

The following video demonstrates the Baxter sorting robot uses the machine-learning perception pipeline to detect cans and bottles and separate them to two different recycle bins:

Sorting bottles and cans using machine learning recognition method

You can read more about this project Here.