Description

In This project, I programmed a robotic arm to "steal" a pen from my hand and throw it on the floor. I used computer vision to detect and locate the pen in the space, and motion planning to control a robotic arm.

Take a look at the project on my GitHub page.

Pen recognition and grasping

Overview

This project contains 2 parts:

1. A computer vision algorithm for recognizing the 3D location of a purple pen, using a D435i depth-camera.
   The software recognize the pen by its color, and locate its coordinate using the 3D camera.
2. Controlling a PincherX100 robotic arm to accurately seize the pen under varying conditions, and throw it on the floor.

Part I: Pen Recognition

In this part, I used computer vision packages (OpenCV) to analyze a video in order to find a purple pen. Then, I used the Realsense 3D camera to locate the pen's coordinates.

The following video is a demonstration of the pen recognition, where the left image is the color frame and the right image is the equivalent depth frame, as captured from the Realsense camera:

Pen recognition

Part II: Pen Stealing

In this part, I used the interbotix motion and control packages to move the arm to a specified coordinates and steal the pen.

The arm motion can be split to 4 steps:

1. Move the arm to the specified coordinates with an open gripper. The coordinates of the pen found using the pen recognition on part I.
2. Close the gripper on the pen.
3. Move the arm to a location above the floor.
4. Open the gripper and release the pen on the floor.

The following video is a demonstration of the pen seizing:

Pen grasping

Similar Projects

In a similar project, I used the same arm and 3D camera to locate, pick, and place objects in the environment, while avoiding surrounding obstacles.

Here is a little taste of this project:

Pick and place while avoiding obstacles

Take a look at this project on my GitHub page.