This paper proposes visuotactile primitives with vibrotactile properties for a more systematic haptic representation of virtual objects in touchscreen mobile devices. We define a DOT, a LINE, and a SURFACE as tactile symbols of hitting a small bump, flicking a string, and rubbing a textured surface, respectively. The combination of the proposed primitives induces a more efficient design process for providing various tactile properties of virtual objects to a user. We did experiments to evaluate whether the proposed method haptically represents virtual objects. The results show that the proposed visuotactile primitives can be applied to the haptic representation of graphical objects.

1. Introduction

The use of touchscreens enables mobile devices to maximize the size of a visual display unit and allows a user to immersively and efficiently interact with graphical contents such as games, maps, and gallery applications as well as a graphical user interface (GUI). Although a larger screen gives users more immersive visual information, it, however, requires considerable cost to enlarge the size of a visual display unit to the level where a user is truly “immersed.” For a more interactive and immersive virtual environment design, we can consider realistic haptic interactions with graphical objects beyond visual and auditory representation. These haptic interactions can be achieved with small vibrotactile actuators or quantitatively controllable vibration modules.

There has been much research on vibrotactile actuators and modules creating haptic sensations in interactive devices. For example, a rotational motor and its speed control can be used for haptic simulation for roughness modulation [1]. An eccentric rotary motor and its sequential control have been commonly used as tactile symbols for information delivery of characters, sounds, and direction [2]. Acoustic speaker vibrators could be used to support communications through mobile devices [3]. Beyond generating tactile sensation in a device, approaches to providing interactive relations between tactile actuators and virtual contents have been investigated with the development of actuators. A good example is mobile phone games with vibrotactile feedback such as bouncing response [4].

With the development of technologies for mobile devices, the user interface of mobile devices has changed from physical buttons to a touchscreen. Touch screens save mobile device space while maximizing the visual display area by superimposing the control surface over the screen [5]. Although larger display area may increase the level of immersion in mobile devices, this technical transition requires new tactile feedback approaches for interacting with graphical elements on a touchscreen. The reason is that it is not easy to increase the size of the visual display unit to the level where users are truly “immersed” by the size of the recreated objects. There have been many research activities on vibrotactile rendering methods which compute the relevant responses of vibrotactile actuators corresponding to the change in the graphic environment. Ahmaniemi et al. proposed a system which creates twelve different virtual textures using a vibration motor [6]. Pakkanen et al. tried to find an optimal haptic sensation when a user pushes several virtual buttons [7]. Hall et al. proposed a small haptic system (a T-Bar) that provides vibrotactile feedback to a user when a user passes the boundary of a virtual object [8]. A vibrotactile rendering method on a mobile platform was studied for an interactive mobile racing game [9]. Martínez et al. suggested a vibrotactile toolkit for haptic interaction with small consumer electronic devices including game devices or mobile devices [10]. Yoo et al. proposed a vibrotactile rendering method based on superimposed vibrations with different frequencies [11]. Safi et al. addressed a vibrotactile rendering method to intuitively access webpages on touch screen devices including mobile devices [12]. Immersion developed the TouchSense haptic enabling toolkit to generate vibrotactile effect by controlling vibration motors [13]. Even though these systems are effective in haptically representing graphical objects, their practical implementation requires two separate approaches which are the design of a graphical environment and the design of appropriate haptic responses. There have been several research works for creating haptic sensation with haptic libraries [1416]. SensAble Technologies Inc® developed the OpenHaptics toolkit which provides low-level function for device access and handles haptic rendering of target objects [14]. SenseGraphics presented a high level scene graph API (H3D API) which is originally web-based technology [15]. Another haptic library (CHAI 3D) was introduced to quickly and easily create visual-haptic scenes [16]. Although these libraries can create various haptic effects from target objects and can convey the effects to users, the created haptic effects are not tactile sensation but kinesthetic feeling. However, many haptic actuators that directly create kinesthetic information are too bulky to be inserted into mobile devices. Therefore, for mobile devices, haptic researchers have focused on stimulating the skin of users. Several studies have shown that tactile interaction is more effective in supporting intuitive operations including gestures in mobile devices [1720]. In order to support designers or contents developers in more effective developments of visuotactile interactive mobile applications, a more general and systematic rendering approach is required.

For a systematic tactile rendering method of virtual objects, we can imagine visuotactile primitives which represent graphical information and tactile information simultaneously. This paper suggests novel visuotactile primitives to haptically render graphic objects with conventional mobile vibrotactile actuators which are already widely embedded in mobile devices.

2. Primitive Model

In this section, we introduce the implementation of three visuotactile primitive models (a DOT, a LINE, and a SURFACE). For the primitive models, we define an Interaction Point (IP) as a virtual location of a finger pad on a touchscreen. When the IP contacts with the proposed primitives on the screen, a vibrotactile response is generated to stimulate a user’s finger. The vibrotactile response for the primitives is determined by a control input, which is a function of the movement speed of the IP and the elapsed time, for a target actuator such as a vibrator.

Figure 1 shows an image set consisting of a DOT, a LINE, and a SURFACE. A DOT is defined as the smallest unit circle with the graphically smallest sized diameter. Its radius is the same as a pixel. The sensation of the DOT is created by providing a pulse signal to a vibration motor so that it can replicate small and bump-like sensations (see (1)). (a normalizing function) normalizes a tactile response value to a range of 0-1, and (a transfer function) converts the normalized value into a corresponding input voltage for a target actuator. The sensation of a DOT is affected by the speed () of an IP movement. While a user rubs over a DOT, an impulse sensation is conveyed to the user according to the exploring speed. If the user rubs the DOT quickly, the duration of the DOT response becomes shorter, and its intensity becomes stronger:where is the input voltage of a target actuator for a DOT, is the impulse between the DOT and the IP, is the time when the IP starts contacting the DOT, is the time when the IP starts escaping from the DOT, and is the virtual mass of the IP.

A LINE model is defined as a unit line segment fixed at both ends; thus, it has a damped oscillation behavior and creates a flicked-string sensation like plucking at the string of a guitar. When an IP contacts a LINE, a static vibrotactile stimulus is conveyed to the user. After releasing from the contact, a damped vibration is generated according to the properties of the LINE during the period from to (the time when the damped vibration converges to zero). The damped oscillation of a LINE is determined with the damping ratio (), natural frequency (), and damped natural frequency (). These are fundamentally settled by the spring coefficient () and damping coefficient () of the LINE. The speed of an IP () also affects the duration and the intensity of the vibration. Equation (2) is a mathematical representation for the LINE model:where is the input voltage of a target actuator for a LINE, is the time when damped vibration is converged on zero, is a clipping function, is the damping ratio of the LINE , is the natural frequency of the LINE , and is the damped natural frequency of the LINE .

A SURFACE model is defined as a unit face with a homogenous roughness. To simulate the roughness of a surface texture, there are periodic small patterns on the SURFACE shown in Figure 2. Because the vibration frequency changes according to the speed of the IP, the SURFACE model is designed based onwhere is the driving voltage of a target actuator for a SURFACE, is the spring coefficient of the SURFACE, is the vertical deformation of a user’s finger, is a clipping function, and is the peak to peak distance (height of the surface pattern).

3. Experimental Results and Evaluation

In the previous section, we proposed visuotactile primitives for the systematic and unified haptic rendering of objects so that users can haptically sense virtual objects which are displayed on the screen of a mobile device. An experiment system was constructed on a mobile hardware platform as shown in Figure 3(a). For creating vibrotactile sensation, we attached a single commercial linear resonant actuator (LRA) [21] to the mobile platform. Furthermore, we used a multipurpose haptic driver which supports an I2C and a serial peripheral interface (SPI) to drive the linear resonant actuator (LRA). For the experiment, a simple application was developed to generate and playback the vibrotactile sensation (Figure 3(b)).

To evaluate whether the proposed method can haptically represent the images, we conducted a matching test. Twenty subjects with an average age of 26.05 years participated in the test. All of them were between 23 and 28 years of age. Seven subjects were female and the rest of the subjects were male. All subjects had experienced mobile platforms and vibration feedback. Two image sets were prepared and inserted into the mobile platform. Each image set consisted of four vibrotactile images shown in Figure 4. After that, we printed two images sets (eight images) onto paper. That is, each sheet of paper contained one image which was printed by a printer. The printed images did not overlap. The vibrotactile images in the first set (Figure 4(a)) have different complexities from each other, whereas the images in the second set (Figure 4(b)) have the same complexity. The term of complexity means the number of objects which can be created by combining more than one primitive. Before the experiment, the subjects learned how to use the developed mobile platform and felt vibrotactile sensations during three minutes.

After basic training, whenever a vibrotactile image was presented on the visual display of the mobile platform in a random order, we asked the subjects to rub the visual display unit to haptically understand without any visual information. After rubbing the unit, the subject watched the printed images and selected one printed image which is the same as the haptically sensed image. During the experiment, each subject experienced all image sets. White sound noise was provided to each subject to prevent the subject from taking a guess due to the vibration sound. To enhance the reliability, this procedure was repeated five times for each image set. Because the number of subjects was 20, the total number of trials was 100. We organized the results of all the subjects in the form of comparison matrixes (Tables 1 and 2). The column of a comparison matrix is the actual images that are presented to the subjects, and its row is the predicted images that the subjects selected. From the results, it is obvious that the subjects were able to distinguish one vibrotactile image from another.

To specify the perception degree of the presented vibrotactile images, we drew confusion matrixes (Tables 3, 4, and 5) which were extracted from the comparison matrixes. In the case of the image set with different complexities (Figure 4(a)), the accuracy is 100%. For the image set with the same complexity (Figure 4(b)), the accuracies are 100%, 97%, 100%, and 97%, respectively (the average accuracy is 98.5%). In addition to the accuracy, we investigated the effect of the complexity on the accuracy by a -test with a null hypothesis: the accuracy is affected by the complexity of the images. Table 6 presents the data set which was used for the -test. The null hypothesis is dismissed at a significance level (, ). The complexity of the vibrotactile images does not affect the perception of the vibrotactile images based on the results. It can be concluded that the proposed visuotactile primitives can haptically represent images effectively.

4. Design Tool

Previously, we developed a design tool on a mobile platform with a touch screen panel for straightforwardly creating, editing, and playing vibrotactile sensations [21]. The developed tool supports two modes: a drawing mode and a playing mode. In the drawing mode, vibrotactile images are selected and displayed on the screen through a touch event handler. In the playing mode, the touch event handler recognizes the touched portion () which is contacted by a user. After that, the collision detector checks the collision between the drawn image and . If there is a collision between the drawn image and , a vibrotactile rendering module produces a control input based on the proposed primitives and conveys the control input to the haptic actuator to create a vibrotactile sensation.

The developed tool supports save/load functions. Vibrotactile images drawn by users can be saved as both a predefined file format and a portable network graphics file format (png file format). We used the extensible markup language (XML) format to implement the save/load functions, and, furthermore, we saved the image data on a secure digital card (an SD card) for portability and compatibility. For more information, refer to our previous study [21].

For easily designing a vibrotactile sensation, we defined a “vibrotactile image” consisting of the proposed primitives (a DOT, a LINE, and a SURFACE) and inserted the vibrotactile images into the design tool. Figure 5 shows the constructed design tool consisting of a drawing area, a draw menu, an edit menu, and a main menu. To maximize the drawing area (Figure 5(a)), we used a drawer menu.

A user selects a vibrotactile image in the draw menu, and, then, he/she drags and drops it in the drawing area. If the user rubs the vibrotactile image, he/she feels a vibrotactile sensation. An interface for editing a drawn image is also provided (Figure 5(c)). The developed design tool consists of three major components: a vibrotactile image editor, a vibrotactile image player, and a data manager. The vibrotactile image editor allows a user to draw vibrotactile images by selecting an item consisting of vibrotactile primitives from a menu. The vibrotactile image player computes haptic information using the proposed primitives. The data manager provides an interface to save and load the vibrotactile image. Figure 6 shows several examples of the vibrotactile images which were drawn by the developed tool.

5. Conclusion

This paper proposed visuotactile primitives which systematically represent graphic objects. The proposed primitives consist of a DOT, a LINE, and a SURFACE corresponding to geometry primitives: a point, a segment, and a plane, respectively. We defined a DOT, a LINE, and a SURFACE as a feeling of hitting a small bump, flicking a string, and rubbing a textured surface, respectively. To show the feasibility of the proposed method, we conducted a matching test with two image sets. The results show that the proposed primitives can haptically represent graphical images regardless of the complexity of the images. Due to the presented primitives, vibrotactile sensation can be easily, directly, and systematically extracted from target objects. Each primitive can be combined into other primitives to haptically render complex shaped objects. Therefore, the presented three primitives and their combination will bring better efficiency on the design process of visuohaptic interaction. Moreover, the suggested concept will facilitate modeling of haptic feedback and rapid prototyping of haptic interaction. Although the presented scheme is so systematic, there is still problem to be solved. It is not easy to apply the proposed primitives to 3D virtual environment which has depth information. In the 3D environment with depth information, two or more different objects can share a pixel. We are currently studying a new method in order to apply the vibrotactile primitives to the 3D environment with depth information.

Competing Interests

The authors declare that they have no competing interests.


This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2015R1D1A1A01059649). This research was supported by a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant no. HI14C0765).