Image processing: table and bottle detection

Lately we've stepped into the computer vision area and worked on the image processing part of the project. The OpenCV library was used.

There are four main tasks that need image processing in real time:

  1. Table detection

      This is the first detection. The robot will localize the table from a distance and get close to it. 

  2. Bottle detection (1)

     The first bottle detection is done while rotating around the table, in order to localize the bottle's position. 

  3. Bottle detection (2) 

      The second detection is done while standing in front of the bottle. This is done mainly because of precision reasons.  

  4. Cup detection

     The last processing is done with the purpose of localizing the cup in which the liquid will be poured.

All of the above mentioned processing steps are done based on template matching, with various improvements within the algorithm.

What have we been up to?

      Up until now we've implemented steps 1 and 3, i.e., the table and bottle (2) detections.

      We have also worked on "centering" the captured image provided that we need this when the robot approaches the previously detected object.

     Apart from this, a LEGO ultrasonic sensor has been installed, which brought improvements to the grabbing of the bottle.

Testing: grab -> lift -> pour

A couple of hours of programming later and we were ready to run a test for grabbing the bottle, lifting it up, pouring liquid and bringing it back to its initial position.

For this, we used:

1.  a Lego motor that controls the gripper (grab)
2.  a Lego motor that lifts up/descends the arm
3.  a Lego motor that rotates the arm (pour)

and 2 Lego touch sensors:

1. to set the home position of the arm for the rotation
2. to signal that the bottle has been grabbed

Below, you can find a demo that illustrates the above mentioned steps:

Side note: for now, the pouring process is in "simulation mode" only :)

More stability, please

While testing the robot's movements, such as plain straight drive, rotations, brakes and arm lifting, we noticed that the robot had the tendency to balance back and forth.

As a solution to this instability problem, we attached a metallic piece to the backside of the iRobot device:

This turned out to be a tremendous help!