Augmented Reality Videoconferencing

What is it about ?

The AR videoconference project is part of the Studierstube collaborative Augmented Reality platform being developed at the Vienna University of Technology. The project integrates the open source telephony library OpenH323 into Studierstube in order to transmit video streams over a TPC/IP-network, allowing for an Augmented Reality videoconferencing system. The videoconferencing module is based on the OpenH323 software, which is an open source protocol stack incorporating a set of communication protocols developed by the International Telecommunications Union (ITU) and used by programs such as Microsoft NetMeeting and equipment such as Cisco Routers to transmit and receive audio and video information over the Internet. The subset of communication protocols we chose relies on the call control and media control protocols and the H.261 video compression standard for low-bandwidth video transmission.
This document describes an Augmented Reality videoconference system, which is a novel remote collaboration tool combining a desktop-based AR system and a videoconference module. The novelty of our system is the combination of these tools with AR applications superimposed on live video background displaying the conference parties’ real environment, merging the advantages of the natural face-to-face communication of videoconferencing and AR’s interaction capabilities with distributed virtual objects using tangible physical artifacts. The simplicity of the system makes it affordable for everyday use. We explain our system design based on concurrent video streaming, optical tracking and 3D application sharing, and provide experimental proof that it yields superior quality compared to pure video streaming with successive optical tracking from the compressed streams. Collaborative AR applications built on our framework are presented, which are followed by the outcomes of an initial user study taking ergonomic and usability aspects into account.

Figure 1. Screenshot of an Augmented Reality videoconferencing session

What do I need ?

The Following list shows you the recommended environment for using our AR videoconferencing tool in Studierstube:

Each user needs:

• a PC connected to a LAN, equipped with a webcam (Firewire or USB)
• the following operating systems:
  - Microsoft Windows 9x/NT/2k/XP
  - Linux
• the latest version of the Studierstube software
  the OpenH323 library
• optical markers (e.g. printed on sheets of paper and glued on a piece of cardboard)

How can I use it ?

1. How to add videoconference background to your applications ?

Proceeding on the assumption that you already have installed the components listed above, this chapter will show you how to configure and run a two-party Augmented Reality videoconference in Studierstube. For each party we need to run two Studierstube workspaces, one for showing the local video and one for showing the remote. Furthermore, for each of the workspace instances we need a different DisplayKit .iv configuration file. You will need to insert two additional lines to these files in order to get the videobackground shown. Let's start by configuring the workspaces showing the local video. The lines we need to add here are identical for both of the parties. Open your configuration file for the local video and add the following lines to the end of the section "display SoDisplayKit":

videoBackgroundOn TRUE
videoBackground SoH323Background{ ipAddress "local" }

The first line containing the boolean variable videoBackgroundOn enables the videobackground, set it to FALSE for no background image. The second line starting with videoBackground defines what kind of background object should be used by Studierstubeviewer. This will be always SoH323Background in our case. The field ipAddress between the braces on the right side of the variable contains the IP address or host name of the party we want to connect to. There are two special values implemented for this field: "local" will show the local video (i.e. the video captured by your own cam). "remote" will wait for a connection and then show the video of the remote party.

Next, let's configure the workspaces for the remote videos. Except for the IP address, this is the same procedure as before. Add the lines  

videoBackgroundOn TRUE
videoBackground SoH323Background{ ipAddress "192.168.0.1" }

to your remote configuration files and substitute "192.168.0.1" for the IP address or host name of the remote party. 

The directory config contains sample configuration files, which you can use, if you are not sure you where to place the lines properly.

2. How to use the prepared sample applications ?

In C:\STB\STUDIERSTUBE\BIN\APPS\VIDEOCONFERENCE you will find some Augmented Reality videoconference sample applications we recommend to try out. This chapter will help you to modify the associated configuration files properly.
At first edit the DisplayKit .iv configuration files in the way we have discussed in the previous chapter. The file videoconferenceUserKit.iv contains the configuration for showing the local video and can be left as supplied. The remote video configuration is defined in videoconferenceUserKit2.iv and contains the IP address of the remote party. Look for the line starting with videoBackground and substitute the IP address of the computer you want to connect to. (Don't forget: You will have to do this for both parties.)
Next, you will have to modify the supplied batch files for running the workspaces. Both files, the one for starting the remote workspace, videoconf_remote.bat, and the one for starting the local workspace videoconf_xxx.bat (where xxx stands for the application name, eg. animal, c3d, objectbrowsing, etc.. ) provide the workspace application with the local IP address right after the command line parameter -smo. Substitute this IP address for your own local IP address.
That's all you need to do. Now you are able to run the videoconference samples by starting the batch files. Have fun !

How does it work ?

This chapter gives you an overview of the implementation of our software and some in depth sight to important details. At the end of it you should be able to change and enhance the source code, make it fit new requirements, for instance move it to a newer version of OpenH323. At first let's have a look at the provided modules. In directory C:\STB\STUDIERSTUBE\SRC\STBAPI\RESOURCE\VIEWER  you will find the following files: 

The following graph shows the inheritance relations between the classes contained in above files:

Figure 2. SoH323 Class hierarchy

On the top right of the graph in Figure 2 you can find SoVideoBackground, the base class for the video background in the SoStudierstubeViewer. (Note for old Studierstube developers: the SoVideoBackground object replaces the old concept of the static OverlayCB function of the VIDEO_OVERLAY.CXX module)
You can use different video backgrounds at runtime without recompiling Studierstube. The usage of a videobackground class is - except for configuration - completely transparent to the Studierstube user and can be easily substituted with another background class. 

Figure 3 contains the definition of the SoVideoBackground class. Beside the standard C++ constructor/destructor couple it defines four virtual methods, which you will have to override in your subclass when you plan to implement a new background object. First are the Open()/Close() methods, called by SoStudierstubeViewer when it activates/deactivates the background image. Both functions have to return a boolean value signaling success or failure of the operation. Furthermore, the Open() method takes two integer parameters specifying the initial width and height of the video image. These values can be changed at any time during a Studierstube session by calling the SetSize() method, for example when the user resizes the Studierstube window. Your code will then have to stretch the background image properly.  And last, the Blittexture() method is called by SoStudierstubeViewer periodically causing the background class to draw the current image (resp. video frame) to the Studierstube window. Since this method is called quite frequently (15 times per second), attention should be payed to optimize it for speed. 

SoH323Background, which is derived from SoVideoBackground, is the most important class of our library, as it implements the interface between Studierstube and the OpenH323 library. It is responsible for managing network connections and displaying incoming video data (local and remote) in the windows of Studierstube at their current size.You have the choice of either displaying the local or the remote video depending on what you've defined in the configuration file. The purpose of each of these classes is best explained considering the following graph in Figure 4, which shows how video data is routed through our objects. 

Figure 4. SoH323 Data stream diagram

Starting on the top left corner of Figure 4, you can see the webcam of Party A sending data to the class SoH323_VideoInputDevice. SoH323_VideoInputDevice grabs the video data of the camera frame by frame by overriding the virtual method GetFrameData() of its base class PVideoInputDevice. It is a device dependent class, in our case we use the ARFrameGrabber library of ARToolKit software package via OpenTracker to obtain the frames, therefore all devices supported by this library can be used. Since the ARToolKit's framegrabber and OpenH323 make use of different video/image formats, we need to convert them appropriately. This is what PColorFormatConverter does. After conversion the image data is handled by the OpenH323 library and routed to the receiving parties. The left side of the graph represents the local video, the right side shows the video transmitted to the remote party. Since both video streams - the local and the remote - are routed through the same classes in the same order, we only need to examine one of them. Let's choose the remote one. In order to be able to keep the required network bandwidth low, OpenH323 compresses the video frames using the H.261 codec. This is a simple, standardized codec working solely on a fixed image size. The different image aspect ratios of our webcam and the image the codec outputs resulted in an undesired black frame around the image edges, which has to be removed at the receiving party. Take care of this if you need to change sizing factors or image flip states. Beside the fact that the OpenH323 project is vaguely documented, the second major drawback of using OpenH323 in this project is the medium quality of the H.261 codec. There is already a better codec on it's way, namely H.263, which is currently available only for Linux but chances are good that in one of the next releases of OpenH323 will include a Windows port, too. After compression the video is sent over the network, decoded on the collaborator's side and frame by frame passed to the SoH323_VideoOutputDevice. The SoStudierstubeViewer object periodically calls the SoVideoBackground::BlitTexture() method to update its background image. Since both the video system of OpenH323 and Studierstube are based on threads and each of them is running at its own rate, we needed a thread-safe interface between SoH323_VideoOutputDevice and SoH323Background. This is realized by the SoH323VideoBuffer class, which manages a buffer holding one video frame and provides methods for locking/unlocking the buffer. Finally the SoH323VideoBackground object resizes and dumps the video images to the SoStudierstubeViewer window. In case you plan to add additional features specific to OpenH323, it's a good idea to study the examples coming with the OpenH323 download package first, especially OpenPhone, which covers the whole range of features OpenH323 offers. At the end of this chapter we present you another graph in Figure 5, which should help you to find object instances more quickly and some tips on compiling SoH323.

Figure 5. SoH323 Object instance relations

Tips for compiling SoH323

There are a few things you should know before compiling SoH323:

Environment variables
There are two system environment variables which have to be set accordingly: 
H323ROOT contains the path to the directory of the OpenH323 library
PWLIBROOT has to contain the path to the PW-Library

Include directories
Take care of the order of the include directories of the PW library, list the os-specific directories first, otherwise you will get compiler errors.

Asnparser tool
The Asnparser utility that comes with the PW library needed to be compiled as release version, the debug version crashed (Windows XP).

***