Augmented Reality on ROS

Table of Contents

  • Introduction
  • Integrating OpenGL in ROS
    • Compiling Customized OpenCV for ROS
    • Editing CMakeLists.txt
    • Testing OpenGL_ROS package
    • Code Explained
    • OpenCV-OpenGL interface
  • Summary
  • Reference

Introduction

If you are working on ROS and want to implement simple Augmented reality applications using OpenGL and OpenCV, this post is for you. The AR applications may have the camera image on the background and drawing a 3D object on the top of the image. In this post, you can see how to draw a 3D object, say a 3D cube on the top of an image. You can apply your own algorithms to move the 3D object based on some object detection algorithms in OpenCV.

Integrating OpenGL in ROS

If you have ROS desktop full installation in your PC, you may have OpenCV installed along with the ROS installation. The OpenCV version comes along with ROS is not compiled with WITH_OPENGL=ON configuration. If it is not compiled with OpenGL, we can’t draw any 3D objects on the top of an OpenCV image.

In order to enable OpenGL on OpenCV which is installed along with ROS, you can use the following method.

Compiling Customized OpenCV for ROS

In order to enable OpenGL in OpenCV, you have to build OpenCV from source compilation.

NoteThe OpenCV version of the source code should be the same as the installed opencv version. Otherwise, the OpenCV-ROS cv bridge will not work properly. In order to avoid ROS-OpenCV compatibility issues, we have to go for the same version of OpenCV. You can use the following script in order to do this task. You can either build opencv locally or you can install it on the system. The safest method is, installing locally and provide the path of opencv in CMakeLists.txt in order to build it from the local version.

You can clone the following OpenGL_ROS package to your catkin workspace in order to get started. The package is also having the shell script to install OpenCV.

cd /<path to your catkin_ws>/src/
git clone https://github.com/qboticslabs/opengl_ros
cd opengl_ros
gedit scripts/install_opencv.sh

Here is the install_opencv.sh script for installing customized OpenCV for ROS.

This image has an empty alt attribute; its file name is 1.png

If you want to install customized OpenCV, first edit the configuration inside this script and save it. The main configuration parameters are;

  • OPENCV_INSTALL: If it ‘YES‘, it will build opencv and install into the system path. If ‘NO‘, it will build locally.
    NoteI will refer you to stick with the local build of OpenCV first because it is safe to work with. Installing OpenCV may cause an undesired effect on your system.
  • WITH_CUDA: If you have NVIDIA GPU and if CUDA installed on your system, you can set the config as ‘ON’. This can accelerate the processing using GPU.
  • WITH_OPENGL: You have to put this configuration ON for compiling OpenCV with OpenGL.

To install the script you can simply follow the command below. Switch to opengl_ros folder and execute the following command

./scripts/install_opencv.sh

It will take from 20 min to 2 hr to build OpenCV. The time will depend on your configuration. Make sure the build is successful before going to the next section. This script will create a folder called opencv_dev_install in your home folder and clone the open cv source code to this folder. If you choose the install option, it may be installed in your system path.

Editing CMakeLists.txt

After building OpenCV, you may have to do some editing in CMakeLists.txt of the opengl_ros package, that you have already cloned in the src folder of your catkin workspace.

I have tested this package in ROS Kinetic and Melodic. The current version of CMakeLists.txt is for ROS Melodic. You can do some simple customization in order to make it compile with Kinetic.

Here are main sections in the CMakeLists.txt

This image has an empty alt attribute; its file name is 2.png

Note: For ROS Kinect users, just search for melodic in the catkin_INCLUDE_DIRS, and catkin_LIBRARIES variable and replace it with kinetic.

You can save the file and build the package in the workspace using the catkin_make command.

catkin_make

If your build is successful, you can able to run the opengl_ros node.

Testing OpenGL_ROS package

In order to test the opengl_ros node, you may use an existing package in ROS. This package will basically open your webcam and publish the images in ROS Topic: ‘/usb_cam/image_raw‘. We can install this package using the following command

Installing usb_cam package in ROS Melodic

sudo apt install ros-melodic-usb-cam

After installing the usb_cam package, you can try the following commands to start the opengl_ros node.

Note: Make sure that you have webcam that is compatible with Ubuntu. Use cheese application in Ubuntu to check the webcam is working.

Launch usb_cam node to start the camera.

roslaunch usb_cam usb_cam-test.launch 

Start the OpenGL-ROS node using the following command

rosrun opengl_ros opengl_ros 

Here is the output video

Code Explained

You can find the OpenGL_ROS code cv_gl_ros.cpp from opengl_ros/src/.

Here is the flow of the opengl_ros node.

This image has an empty alt attribute; its file name is 3.png

Acquiring ROS Image messages

If you check the cv_gl_ros.cpp, you can find a class ImageROS, which helps to receive and publish ROS images.

This image has an empty alt attribute; its file name is 4.png

Initializing track bar and OpenGL rendering Window

You can see the initialization of the OpenGL render window and two trackbars. One is for changing the x_rotation and others for changing y_rotation of the 3D cube.

This image has an empty alt attribute; its file name is 5.png

Render 2D texture and 3D object

The on_opengl(void* param) is a callback function that is responsible for drawing the 2D texture in the background and drawing a 3D cube on top of it. In this code, the 2D texture is the input image itself. The callback read the trackbar position and render the cube orientation based on that value.

This image has an empty alt attribute; its file name is 6.png

Here is the while() loop which is doing the image acquisition, 2D and 3D rendering in OpenGL window, and checking for ‘Esc‘ key in order to quit the program.

This image has an empty alt attribute; its file name is 7.png

OpenCV-OpenGL interface

You can find the existing integration of OpenCV-OpenGL APIs from the following link.

OpenCV-OpenGL API Reference

Summary

In this post, you have seen how to create simple Augmented reality using OpenGL, OpenCV, and ROS. We have seen how to enable OpenGL on OpenCV which is installed along with ROS. We have seen a script to install OpenCV by source compilation and have discussed how to customize in order to compile with custom OpenCV, ROS Melodic, and Kinetic. After that, we have tested the opengl_ros node and finally, you have seen a brief explanation of the code.

Reference

What is ROS?

R.O.S stands for Robot Operating System. It is not a real operating system. ROS is called a meta operating system, it is having some features of a real operating system. It is a Free & Open Source Robotics Software framework mainly for programming complex robots.

ROS was originally developed in 2007 by the Stanford Artificial Intelligence Laboratory. As of 2008, development continues primarily at Willow Garage. From 2013, ROS development and maintenance are doing by Open Robotics.

It is a great software tool for quickly prototyping a robot software, without the need of writing everything from scratch. We can even think of using ROS on the real robot products but it may require extra refining of the prototype code and should keep code quality standards.

Here is the ROS Equation given on the ROS website.

ROS consists of four main parts. The ‘Plumbing‘ is an important part of the ROS.

Plumbing

The ‘Plumbing‘ can be also called communication middleware, which helps two or more programs to communicate with each other. The communication middleware provides APIs to us to send & receive various types of data between multiple programs running on our computer.

In short, the Plumbing part in ROS creates a network of programs/nodes which are communicating with each other by sending & receiving different kinds of data. We can call this as a ROS Computation graph.

Visualization of a typical ROS Computation Graph

When we work on a robot, we may have to write many programs to handle sensor data and control the motors of the robot. The ROS plumbing helps us to create a computation graph to manage robot software.


Tools

The Tools present in ROS helps to debug and monitor various kinds of data that are sending & receiving by ROS nodes. There are Graphical tools and command-line tools present in ROS. In a robot, there are various sensors such as a camera, Laser scanner, etc. Using ROS tools, we can visualize various kinds of sensor data and debug it. Examples of ROS tools are Rviz and Rqt.


Capabilities

Using the ‘plumbing‘ feature in ROS, many robot software blocks are built for doing various capabilities in robots such as navigation, perception, manipulation, etc. If someone wants to implement these capabilities in their robot, they can simply re-use the software from ROS and save time.


Ecosystem

The ROS open-source robotics framework is powered by thousands of developers across the globe who are contributing and maintaining thousands of ROS packages, tutorials, Q&A, etc. There is a vibrant community of developers and users around the globe that makes ROS as a unique framework in Robotics.

ROS Developers across the globe, Courtesy: ros.org

ROS is one of the software frameworks used in most of the universities for robotics research. Other than academic research, ROS is also used in robotics companies to prototype their software. ROS is also used by hobbyists to create different robot applications.

If you are a roboticist or aspiring roboticist, ROS will be a great tool to program robots. Knowing ROS doesn’t mean that you know all about robotics, but instead, it will be a useful tool to prototype your robotics algorithms.

There are great opportunities for ROS programmers in academic research as well as corporate, so learning ROS will be a great addition to your skills to pursue a robotics career.