SignPost - Mobile AR Navigation System

SignPost is a Studierstube application that is able to guide a person through an unfamiliar building. The person wears a mobile navigation equipment that consists of:

a head mounted Glasstron optical stereo see-through display a helmet with an Intersense InterTrax2 inertial tracker a FireWire camera a backpacked notebook computer with an accelerated 3D graphics chip a wrist pad that allows user input

The see-through glasses enable the user to see the real environment of the room in which he is currently standing. SignPost allows the user to select a destination within the building and guides him or her to the desired destination by overlaying hints (i.e. direction arrows and a wireframe model of the room and its portals) on the user's view of the scene through the glasses. The application searches the shortest path starting from his or her current position and directs the user along that path to the selected room.

To determine the user's current position, SignPost of course needs some sort of tracking.
As we cannot rely on GPS or similar positioning systems inside a building, tracking of the user is done with ARToolkit markers and the FireWire camera in combination with the Intersense inertial tracker which is particularly dedicated for fast changes in the orientation of the user. Details of tracking are described below.

Interaction

As mentioned before, the user wears a wrist pad, that allows him or her to interact with the SignPost navigation system. Whenever the wrist pad is brought into the user's view a WIM model ("world in miniature") is overlaid onto the scene, providing the person, who wants to be guided through the building, with an overview of all rooms on the floor. The user's current position is displayed as a yellow room, whereas the destination room is shown in red. The path to the destination is shown is highlighted in cyan.

At startup of SignPost, the application (of course) has no knowledge of the room in which the user is. The current position is either determined by viewing a unique marker (described in section 'Marker re-use') or by selecting the current room on the wrist pad. Then the person can select the desired destination. The wrist pad provides the following options:

After selecting a destination room, SignPost queries its own Pathfinder-component to find the shortest path from the current position to the selected destination. The Pathfinder relies on an adjacent graph, which is built at startup from the floorplan file (see below), and returns the distance of the shortest path as well as a list of the portals and rooms, which have to be passed on the way to the destination room. Based on that path and the geometry information of the floorplan, the SignPost system guides the user through the building by providing him or her hints on the see-through display.

Navigation Hints

These are the navigation hints that are overlaid on the user's view through the glasses:

	The WIM (world in miniature) model is attached to the wrist pad and always shows a full view of the floorplan, with the person's current position (yellow), the path he or she has to go (cyan) and the destination (red). The WIM model does not permanently cover the screen; the user can bring the WIM into his view by moving his arm with the wrist pad in front of his eyes.
	Forward arrow: The next door or portal is straight forward.
	Left arrow: The user should turn left to find the next door or portal.
	Right arrow: The user should turn right to find the next door or portal.
	U-Turn: Wrong direction - the user has to turn around to find the next door or portal.
	Wire frame: The three-dimensional geometry information of the rooms and portals derived from the floorplan file can be displayed as well. The edges of the room geometry should be aligned with the edges of the real environment. As there sometimes appear jitter or large displacements in the overlay, the wire frame is optional and can be turned on and off by a button on the wrist pad.

Floorplan

SignPost relies on a floorplan, that holds three-dimensional geometry information of the rooms within the building, as well as information of ARToolkit marker placement (see 'Tracking'), in order to allow the application to determine the current position of the user within a room.

SignPost (as well as Studierstube) works with OpenInventor scene graphs for displaying 3D geometry. The geometry of the rooms and portals is directly loaded from an OpenInventor (.iv) file, which makes changes and extensions to the building information very flexible. However, data acquisition was very work intensive, as for each room and portal several coordinates have to be provided (plus the offset of the room within the floor) and additionally the position as well as the orientation of a lot of ARToolkit markers has to be measured and stored. Thus, editing all this information in a single textual file is not very comfortable.

Map Viewer To overcome this problem, we think about an XML file format to make the building geometry information more structured. Also, adjacency information - e.g. which room has a portal to another room - could be saved more logically. However, there would still be a textual file that someone has to edit.

The most convenient solution will be a Map Editor that allows graphical interaction, such as interactive room and marker placement with the mouse. Until then, we provide a Map Viewer tool, which shows the currently stored floorplan together with the marker positions, their orientation and ID numbers. The Map Viewer enables controlling the information stored in the floorplan file; it allows zooming as well as three-dimensional rotation of the building geometry. Consequently, someone who does changes or extensions to the floorplan can immediatly check, whether his or her changes make sense and if everything is placed on the correct position. Without this tool, errors in the floorplan may hardly be found in the SignPost application and there may be strange results while guiding a user within the building ...

Tracking

Tracking of the person who uses the SignPost navigation system is done by a combination of optical and inertial tracking. In all rooms and hallways, which are covered by our floorplan, ARToolkit markers were mounted on the walls: at least one marker per wall, on larger walls one marker every 2 - 3 meters. An ARToolkit marker consists of a black square frame filled with some kind of a pattern. The patterns must have no rotational symmetry, have to be sufficiently distinct and have to be trained with ARToolkit in order to be recognized. ARToolkit - the Augmented Reality Toolkit - is an external component, which provides SignPost (through OpenTracker within the Studierstube framework) with optical tracking data. The FireWire camera that resides on the user's helmet delivers a video stream to ARToolkit, which detects the markers in the video image. According to the size and orientation of the marker on the video image, it is able to compute the relative position of the user in respect to the marker. By combining the relative marker location and the positional and geometry information given in the floorplan file, the user's movement within the environment can be tracked with certain accuracy.

The Intersense inertial tracker, also mounted on the user's helmet, is able to track the user's orientation. It provides more frequent updates than the ARToolkit tracking and therefore is optimal for rapid movements of the user's head. Besides, it is also suited for filling gaps in tracking, e.g. when optical tracking failed or no marker is visible in the user's line of sight.

Marker re-use

In order to allow reasonable tracking, ARToolkit markers have to be placed every 2 - 3 meters. Deploying markers on our floor, which covers about 25 rooms and long hallways, would require several hundred different marker patterns to be created and trained for ARToolkit. The more markers we use, the higher the degree of similarity will be. A large set of markers also increases the search space that has to be compared by ARToolkit. Consequently, a large amount of markers leads to a decrease in performance and/or a higher number of false recognitions.

To overcome this problem, we developed a marker partitioning-scheme that allows re-using sets of marker. The idea is, that if the tracking system knows the user's current location, it can rule out large parts of the building because they will not be visible to the camera. Thus, for two areas, which are not visible to each other, it becomes possible to use the same set of markers. Based on the room models in the floorplan file, the application builds an adjacent graph using the rooms as nodes and the portals between rooms as edges. In addition to the portals representing logical connections, further edges that represent a line-of-sight between two rooms are added. An example of this adjacent graph is depicted on the left.

With that approach, we can generate a minimal number of disjoint sets of markers. Two rooms with a connection in the adjacent graph described above must have disjoint sets of markers. Otherwise the camera tracking system cannot decide which room a certain marker belongs to, if both instances are visible from one room. Rooms that have no connection in the adjacent graph can use the same set of markers (i.e. markers can be re-used).

This approach improves performance in two ways: 1) The overall amount of markers is minimized, which reduces false recognitions and minimizes the search space of ARToolkit. 2) Starting at a given room position, the system has only to track markers assigned to the current room and the neighboring rooms. Thus, a lot of markers are ruled out before the recognition process.

However, the marker re-usage leaves one question open: How does the application determine the initial room, where the user starts the application? For this purpose, we created an extra set of markers - the unique marker set. In every room, one marker of these unique markers is placed and assigned to that room. As a result, the application can determine the initial room immediately after recognizing the room's unique marker - all the user has to do is just to turn around 360 degrees once.

Testing & Future Work

The current model used for development and testing purposes covers parts of two adjacent floors in our institute building. The floorplan file holds geometry and adjacency information for 22 rooms. A total amount of 40 markers was used. Initial experiences showed that the system is heavily relaying on sufficient marker density. Whenever markers are too far away (or too small) much jitter is perceived by the wireframe model. Note, that the wireframe model can be turned off by the user, and the jitter does have nearly no impact on the direction arrows that are displayed to the user, because they still point to the next portal the user is meant to go to. It can be expected, that the jitter is reduced with a larger marker density. Gaps, which sometimes occur between markers, are mostly bypassed by using the inertial tracker. Unfortunately, the inertial tracker sometimes starts to drift in one direction leading the user to bear away from his path. Heading to a wall and viewing a marker from a close position adjusts the path shown on the users' view to the correct path again.

We found, that the degree of correct marker recognition relies heavily on the light conditions in the rooms. ARToolkit provides a useful feature to address this problem: One can assign a threshold to marker recognition, which influences the lightness at which a marker is recognized properly. However, in our application we are confronted with varying light conditions from one room to another or even in a single room.
We plan to implement adaptable thresholds, which allow to assign each room its own threshold for marker recognition according to its prior-known lighting.
Also, the quality of the tracking system could be improved by using Kalman filters to predict the user's motion.

We want to extend our digitized map to all floors of our building, as a more complete model would of course allow more comprehensive location based services. For this amount of digitalization professional grade surveying equipment will be necessary, but we think the result will justify the effort. The setup is suitable for outdoor scenes as well, and we intend to try this soon.

Material

Video

The demonstration video shows a user, who is guided by SignPost from the Virtual Reality Lab on 4th floor of IMS institute to the library on 5th floor. You can download the video from here:

Paper

SignPost and the marker re-use scheme were presented at ART02 - the First IEEE International Augmented Reality Toolkit Workshop. The publication 'Structured Visual Markers for Indoor Pathfinding' is available from here:

• read the abstract
• download PDF document

Online slides

Slides of a presentation of an early version of SignPost can be viewed here:

• online slides

Credits

The SignPost application, the MapViewer tool, the ART02 paper, the SignPost and 'Structured Visual Markers for Indoor Pathfinding' presentations and this Webpage ...

... were developed, written & produced by these people: Michael Kalkusch Thomas Lidy Michael Knapp Gerhard Reitmayr Hannes Kaufmann Dieter Schmalstieg

Michael Knapp, Thomas Lidy, Michael Kalkusch