common tasks in desktop information visualization tools, such as data dynamic filters, attribute selection .... attribut
Multidimensional Information Visualization Using Augmented Reality Bianchi Serique Meiguins 1 , Aruanda Simões Gonçalves2 , César Siqueira de Oliveira 1 , Ricardo Melo Casseb do Carmo 1 , Sérgio Clayton V. Pinheiro 2 , Leonardo Hernandez1 1
2
Departamento de Informática – Universidade Federal do Pará (UFPA) 66.000-000 – Belém – PA – Brasil
Área de Ciências e Tecnologia - Centro de Universitário do Pará (CESUPA) 66.000-000 – Belém – PA – Brasil
[email protected],
[email protected], {cesarso, rcasseb}@ufpa.br,
[email protected],
[email protected]
Abstract. This paper presents a prototype for information visualization in an Augmented Reality environment. The prototype allows users to perform common tasks in desktop information visualization tools, such as data dynamic filters, attribute selection, semantic zoom and details on demand. Such tasks are seldom available in AR environments. We used a customized version of ARToolKit, a software tool that allows the insertion of virtual objects in real environments through cards with marks. The interactive controls as well as the data visualization are projected to the user and manipulated through these cards. Resumo. Este artigo apresenta um protótipo para visualização de informações em ambiente de Realidade Aumentada. O protótipo permite aos usuários realizarem tarefas comuns em ferramentas de visualização de informação desktop, tais como filtros dinâmicos de dados, seleção de atributos, zoom semântico e detalhes sob demanda, que em ambientes de RA quase nunca são vistos. Para a concepção do protótipo foi utilizada uma versão modificada do ARToolKit, software que permite incluir objetos virtuais em ambientes reais através de cartões com marcações. Os menus de comandos e controles interativos virtuais são projetados e manipulados através destes cartões, bem como a visualização dos dados para o usuário.
1. Introduction The ability to evaluate a decision- making process and perform rapid and accurate strategic changes is each day more important, specially on the fields of industry, economics and finances. In order to support theses processes, Data Mining, Information Visualization and Intelligent Systems projects are getting more and more attention from the research community. Specifically in the area of Information Visualization, new tools and techniques are constantly being developed to cope with the growing demand. An Information Visualization tool has the purpose of creating an interactive visual representation that transforms abstract data in a way that may be promptly understood by the user and may be used for tasks such as identification, multivariate correlation, search, exploration and communication.
Many current computational systems have tried to get more adaptable to human perceptions through more interactive user interfaces. This is also an important issue when it comes to Information Visualization tools. In this context, Augmented Reality (AR) studies new interaction and visualization mechanisms and allows a more natural user-to-system and user-to-user communication. AR creates enriched real environments by merging to this predominantly real world objects as geometric models, images, sounds and text and improving user perception. AR is a multidisciplinary field and may be adopted by Informatio n Visualization tools. The goal of this paper is to present a Visualization Information in Augmented Reality prototype where the user may visualize and manipulate information in a real time three-dimension environment without the use of devices such as a keyboard or mouse and interact simultaneously with other users in order to make a decision related to the analyzed data. The contribution of this prototype are the new interaction mechanisms, the use of interactive menus and the implementation of filter techniques that modify the environment in real time without modifying all virtual objects. This allows the prototype to be more efficient for the support of a decision- making process. The prototype uses of ARToolKit, a free open-source library for the development of augmented reality applications, with the addition of new functionalities. OpenGL was used for the creation of virtual three-dimensional objects. This paper is organized as follows. Section 2 briefly presents Information Visualization concepts, characteristics and main techniques. Section 3 defines Augmented Reality and details of ARToolKit, the tool used for the construction of the prototype. Section 4 presents detailed development issues of the prototype. Finally, section 5 presents final remarks and future work proposals. 1.1 Related Works Bueno (2005) extended the Meta3D tool creating a AR module for Information Visualization. The user interaction in this tool depends on the configuration interface in a 2D environment and the data are grouped in clusters each cluster associated to a marker. The information filter quality depends on the number of markers. The tool used Chernoff Faces and Extended Parallel Coordinates techniques. Buk (2005) presented AR as an alternative for information visualizatio n, where graphics are presented over real- world objects. However, filters are also associated to makers making the configuration options less flexible. The setting mechanisms are also 2D. Slay (1994) uses AR to visualize the Graph-based data representation technique. The setting options interface and the view generation is 2D. The intention of building a virtual marker from VRML objects is remarked. The tool does not present filter techniques on the AR interface. ARToolKit was used in all the above projects. An aspect in common is the absence of efficient filter techniques within the augmented interface.
2. Information Visualization In the different areas of human activities, the arduous task of extracting knowledge from data sources is primordial to improve the decision- making process. Professionals who are responsible for making decisions need efficient tools to help them perform their tasks in a very fast, simple and precise way. The Information Visualization area fulfills these requirements, since Informa tion Visualization is, by definition, the process of turning abstract data into a visual shape easily understood by the user, making it possible for him/her to generate new knowledge about the relations between the data [Spencer 2001]. The visual form in which information is available to the user has influence on the knowledge extraction task of an information system and depends on the data. Data can be classified in 7 (seven) types and to each one of them a different visualization is described [Sheneiderma n 1996]. The data types are: •
1-D: Type of data represented by texts.
•
2-D: This type of data is used to represent geographic maps, engineering plants, etc.
•
3-D: This type of data has the characteristics of bi-dimensional data plus the information of volume.
•
Multidimensional: Multidimensional data describes an item with more than three attributes. Dynamic consulting techniques [Card 1999] and dispersion diagrams are used to improve visualization of this type of data.
•
Temporal: Type of data where the attribute is incorporated to previously described types.
•
Hierarchic: Type of data which has a hierarchic structure where each item has a parent node.
•
Network Data: nodes connected by links previously defined. Theses links can be organized in trees or hierarchically and the best manipulation way is to allow changes in node focus.
According to Carr (1999), an information visualization tool should allow users to perform the following tasks: •
General View: the user needs an overall view of all analyzed data, based on the selected parameters for the visualization
•
Zoom: the zoom technique is important because it allows focusing on the analyzed data. Semantic zoom also allows the user to visualize more details to the visualization.
•
Filter: users frequently need to reduce the size of data sets by reducing the number of attributes. One of the most efficient ways is the use of Dynamic Queries, which is basically a technique that allows the user to select data without the use of any kind of command lines.
•
Details on demand: when users are exploring a data set, they’ll need to see details about one item in particular.
•
Relationship: If the user discovers an item of interest, he/she might need to know about other items with similar attributes, so the tool could point out these similar items.
•
Historic: the user needs support to undo an action, show the steps until that point, etc.
3. Augmented Reality Augmented Reality’s main characteristic is the creation of enriched real environments, adding to this predominantly real environment items such as geometric models, images, sound and texts and potentializing the utilization of the user’s senses for a better perception of the environment. This way, Systems in AR generate to the user a combined vision of virtual objects generated by computer with real objects visualized by the user, enlarging the scene with additional information [Vallino 1998]. Augmented Reality is a particularization of a broader area known as Mixed Reality, which incorporates “virtual elements to the real environment (Augmented Reality) or takes real elements to the virtual environment (Augmented Virtuality), completing the environments” [Milgram and Kishino 1994]. One of the great advantages of using AR is the possibility for users to interact with the virtual and real worlds integrated. According to the type of display that is used for visualization and projection of virtual objects in the real environment, Augmented Reality can be classified in four main groups [Milgram and Kishino 1994]: •
Optical See-Trough AR : Combines virtual objects directly in the real scenario through Virtual Reality semi- transparent glasses.
•
Video See-Trough AR : Opaque Virtual Reality helmets are used, with the help of a video camera that captures the real images and incorporates them with to the virtual environment.
•
Monitor Based AR: Presents a mix of videos captured from real environments and virtual scenarios in monitors of computational devices.
•
Projector Based AR : Projects the virtual environment in surfaces of real objects.
3.1. ARToolKit Developed by HIT Lab and distributed as an open code written in C Language. ARToolKit is a tool that allows that programmers develop applications in AR [Kato et al. 2003]. This library uses computational vision techniques to precisely calculate the position and orientation of a camera related to a marker in real time. The programmer can use this information to draw 3D objects exactly lined with the real ones. Figure 1 illustrates a basic cycle of ARToolKit execution. At first, an image of the real world is captured by any video input device (a webcam, for example) and then transformed into a binary image. The binary image is searched for squared regions. The next step is to calculate the position and orientation of the camera in relation to the squared regions that represent s possible cards containing specific symbols called markers. The markers should contain different symbols previously registered through a
training of the internal neural net of ARToolKit for effective recognition. Once the marker is recognized, the next step is performed, when the tool calculates the exact point the virtual object must be in the real world builds an image and returns a visual combination of the real world and the virtual object to the user.
Figure 1 – ARToolKit Execution Steps
The construction of objects to be combined to real world can be done through applications in OpenGL and VRML. There is also a version of ARToolKit written in Java (JARToolKit) [Geiger 2005] where JAVA3D can be used [Wash and Gehringer 2002].
4. The Prototype In this section, the prototype that uses augmented reality to implement information visualization techniques is presented. The user interaction occurs in a very direct way, manipulating filters and 3D objects with his/her own hands instead of using conventional devices such as mouse and keyboard. It is also possible to manipulate virtual objects to select other virtual objects. The prototype produced modifications in ARToolKit and in OpenGL.
Figure 2 – Overview of the Prototype Architecture
Figure 2 presents a general view of the changes or new implementations that were made in ARToolKit to develop the prototype presented in this paper. Some modules stand out: Interaction detection module, where the prototype identifies the action the user wants to perform; 3D objects settings module, which receives information about an action of the user and modifies the 3D objects that will be
presented to the user; and the database reader module, which is responsible for loading the data to the memory so that the prototype can manipulate the data. The main phases that must be accomplished in order to develop a good visualization project are the following: •
Define with the user the data items that are relevant to the visualization project;
•
Previous processing of the selected database;
•
Identify the data types available in the base and their dimensionality;
•
Identify a group of IV techniques that properly represent the data;
•
Identify the best way to represent the data;
• Identify the best way to manipulate the represented data. 4.1. Database An important pre-requisite in the implementation of the visualization technique is the initial treatment applied to the database. During this phase, the use of attribute selection techniques is recommended to reduce the volume of the existing data, using only the attributes that contribute to a good data analysis. This preparatory phase demands a high level of knowledge of the information that is being worked with to prevent important attributes that are initially considered irreleva nt from being eliminated. The prototype used a public domain database containing information on Brazilian automobiles. After the preparatory phase, the final table consisted of 41 lines (including the name and type of attributes) and 10 columns: brand, model, fuel, year, color, price, number of doors, type, power and r.p.m. A small sample can be seen in Table 1. Table 1 – Brasilian Car Data MAKE
MODEL
FUEL
YEAR COLOR PRICE DOOR
TYPE
POWER RPM
STRING
STRING CADEIA INT
STRING REAL
INT
STRING INT
INT
Fiat
Uno
1993
prata
6900
4
hatch
55
4000
Kia
Sportage diesel
1995
prata
40000
4
pickup
120
4000
1993
branco
9400
2
wagon
99
5600
volkswagen Parati
gasolina gasolina
4.2. Visualization Techniques The 3D Scatterplot is the main technique used to visualize the objects. In this graphic model, the 3D objects are positioned according to x, y and z axis. Each visualized object has specific characteristics of color, shape and size that directly represent data items values, in order to magnify the perception of the data. This characterizes the implemented technique as multi-dimensional (information of x, y and z axis, color, shape and size). In the initial settings (Figure 3), the x axis is set to the year attribute, the y axis to the automobile price, the z axis to the automobile power, the color represents the brand, the size the number of doors and the shape represents the type of car.
Figure 3 – Initial settings
In Figure 3, the objects circled in red can be interpreted by the caption beside the graphic. The big pink circle is interpreted like Sedan – Honda – 4 doors and the small one as Sedan – Honda – 2 doors, the big red cylinder as Hatch – Ford – 4 doors and the small one as Hatch – Ford – 2 doors. 4.3. Augmented Graphic Interface The graphic interface consists of continuous and discrete filter controls and 3D scatterplot projected on marks. Each marker position was carefully determined to avoid frequent ARTookKit manipulation mistakes, such as undesired occlusion of markers. The group of markers and their functionalities is illustrated in Figure 4.
Figure 4 – Distribution of Markers
Figure 4 illustrates the real position and the virtual content of each marker used in the augmented interface. An example to filter the elements in the “cone-shaped” visualization is underlined in Figure 4. In order to do so, the user must fisrt choose an item from the Main Menu. In the example, the item must be “Shape Filter”. The item selection is done by blocking marker 1. Everytime the marker caption from the camera is blocked, a new item from the Main Menu becomes available in the sequence illustrated in Figure 4. The other markers are automatically reconfigured to attend to each sub-item from the Main Menu. The next step for the user is to select the shape of the visualization (cone, cube, sphere, and others), and it will be able to be seen through marker 3 and chosen through marker 2 or marker 4. Then the user should select “Hide” or “Isolate” the objects with the geometric shape selected and confirm the action with the “Yes” command, using marker 6.
Figure 5 – Creation of Dynamic Charts using Markers
During the execution, the user’s interaction with the graphic interface of the prototype happens in a direct way, occluding the marker where the desired virtual control is projected for a certain action to be executed. The interaction by occlusion consists in obstructing the capture of a marker by the webcam. The obstruction can be done by any kind of material capable of overlapping part of a marker symbol. A good suggestion is the use of one’s own hands to perform this action. 4.3.1 Using Filter The developed filters were based on the dynamic queries concepts and support both discrete and continuous attributes. Dynamic queries allow the users to perform queries in a database without the need for command lines, using exclusively graphic interface components [Sheneiderman 1994]. The discrete (or nominal) attributes are usually used as setting options such as color, size and shape. Figures 6 and 7 illustrate the use of the shape filter, isolating only one shape (Figure 6) or removing that shape from the view (Figure 7). Continuous attributes are excellent candidates for axis representation. However, every continuous attribute has a related interval selection control, whether or not it is mapped to one of
the axis(Figure 8). The prototype presents graphically the filter operations results in real time.
Figure 6 – Discrete attribute filter – isolating Shape.
Figure 7 – Discrete attribute filter – hiding Shape.
Figure 8 – Continuous attribute filter – Power.
4.3.2 Details on Demand There are three forms of representation available: color, shape and size, but there is still more information to be analyzed about each object. A virtual mechanism, a “virtual pointer”, was developed to select a virtual object and present extra information (Figure 9). The virtual pointer is a virtual object also based on markers. In order to move it, the marker to which it’s associated should be moved. Its goal is to select virtual objects in the visualization and it uses the concept of intersection.between objects to do so. At the end of the virtual pointer there is a spherical object that, when in contact with another virtual object, can take the shape of wireframe, identifying the object that was identified or selected. The virtual pointer uses red to identify a selection.
Figure 9 – Retrieving information on an object.
Figure 10 – Semantic Zoom for the attribute Model.
4.3.3 Semantic Zoom The prototype already presents zoom, rotation and translation mechanisms. They are available on the menus which are appropriate to virtual and augmented reality applications. Additionally, the user may analyze the chart freely with the marker in his/her hands. Semantic zoom allows the user to observe the virtual objects in greater detail as his/her view approximates the objects (Figure 10). 4.3.3 2D Charts The 2D charts help the user providing new information on the visualized data. There are two types or charts available: histogram and pie. The pie chart for the attribute Fuel may be seen in Figure 11.
Figure 11 – Pie chart for the attribute Fuel.
Figure 12 – Use of the Help menu item.
4.3.3 Help The help item presents the user in detail the operation and content of each marker. The use of help is very easy. With the use of the virtual pointer, the user selects the help option first. The next step, the user should then select one of the menu items and read the available information about it. In order to disable the help command, the user must select the help option again (Figure 12).
5. Final Remarks and Future Work This paper presents a prototype for a multidimensional information visualization tool in an augmented reality environment. The AR environment provides a more intuitive and motivated interaction for the user. It also allows the user to manipulate the environment while interacting with other users in a collaborative fashion. The prototype aims to satisfy the main characteristics of a good visualization tool: overview, filter, details-ondemand and semantic zoom. The input is currently supported as text files but the development was generic enough to be adapted to any relational database. Other remarkable aspect is the implementation of menus in the mark cards. This allows a larger set of func tionalities with the use of a lower number of markers. It is possible to set in real time axis, color, size and shape parameters, generating many views for the same environment. The use of dynamic queries allows the user to generate any combination of queries and filters without the need for command lines. The prototype also allows the use of 2D graphics to provide complementary information about visible data. Additionally, two implementation aspects contributed to make the interaction with the 3D virtual objects a lot easier: the virtual pointer and the help about virtual components in the AR interface. The main difficulties in the development of the prototype were: the instability of ARToolKit, specifically in environment lightening and marker detection; the comprehension of API such as OpenGL, especially when it comes to sequences of transformations; and the need for advanced hardware – a 128mb video card, for example. As future work we intend to refine and optimize the code for publication in a website. It is also our intention to use a data glove to manipulate the objects and use the marker only to include 3D objects on the real scene. Incorporate the support for database access and generate multi- user collaborative version are also among our future goals. It could be studied the addition of another webcam to enlarge the user eye vision using two video monitors. Finally, a next step will be to perform usability and ergonomics tests comparing with similar desktop tools.
References Bueno, et. al. “Meta3D++ - Visualização de Informações em Realidade Aumentada”, WRA 2005, Piracicaba, Anais. Piracicaba: UNIMEP, 2005. p.33-36. Buk, et. al. “Visualização de Informações em Ambientes de Realidade Aumentada”, WRA 2005, Piracicaba, Anais. Piracicaba: UNIMEP, 2005. p.68-71. Card, S., Mackinlay, J., and Shneiderman, B. eds. Readings in Information Visualization Using Vision to Think. Morgan Kaufmann, 1999. Carr, D. A. Guidelines for Designing Information Visualization Applications. Proceedings of ECUE’99. Estocolmo, Suécia. Dezembro 1999. Geiger, Christian. (2005) “JARToolKit – A Java binding for ARToolKit”, IEEE JARToolKit. Available: http://jerry.c- lab.de/jartoolkit/. Access in: 11/15/2005. Kato, H., Billinghurst, M., Poupyrev, I. (2003) “ARTolKit version 2.33 Manual”, Nov.,2003. http://www.hitl.washington.edu/research/shared_space. Access in: 11/20/2005.
Milgram, P. e F. Kishino (1994). “A Taxonomy of Mixed Reality Visual Displays.” IEICE Transactions on Information Systems. Sheneiderman, B. “The eyes have it: a task by data type taxonomy for information visualizations”, Procedings of 1996 IEEE Visual Language. 336-343. Shneiderman, B., Dynamic queries for visual information seeking, IEEE Software 11, 6 (1994), 70-77. Slay, H., Philips, M., Vernik, R., Thomas, B. Interaction Modes for Augmented Reality Visualization. Proceedings of Australian Symposium on Information Visualisation, Sydney, December 2001. Spence, Robert. Information Visualization. Addison-Wesley.(2001). Vallino, J. “Interactive Augmented Reality”, Department of Computer Science, University of Rochester, New York, 1998. Walsh, A., E., Gehringer, D. Java 3D API Jump -Start. Prentice Hall PTR. (2002).
