Abstract
Previous augmented-reality (AR) applications have required users to observe the integration of real and virtual images on a display. This study proposes a novel concept regarding AR applications. By integrating AR techniques with marker identification, virtual-image output, imaging, and image-interaction processes, this study rendered virtual images that can interact with predefined markers in a real three-dimensional (3D) environment.
1. Introduction
Recently, augmented-reality (AR) techniques have been widely applied in modern technology, allowing developers to incorporate computer-generated virtual images into actual footage that was recorded using a camera. Thus, AR is observed in numerous applications and fields [1–7] such as advertising, films, visual conferences, education, and smart phones.
Previously, integrated real-space and virtual images using AR techniques could only be viewed on a display [8]. For example, Milgram proposed a see-through and monitor-based AR display method that required a display to enable interaction and defined markers and virtual image as indirect interactions. Recent AR products, such as Google Glass, are also see-through methods. The AR system developed in this study differs from those methods because this system enables direct-interaction functionality. This functionality is achieved by using a system-imaging device, which renders virtual images that overlap predefined real space. In addition, the proposed system is in compliance with the three characteristics proposed by Azuma, who stated that AR applications must combine real and virtual elements, be interactive in real time, and operate in three-dimensional (3D) environment [9].
2. The Direct-Interaction 3D System Design
In the application process of AR techniques, this study designed a direct-interaction system that enables direct interactions between real and virtual environments, as shown in Figure 1. The figure depicts a user holding a marker that is used to interact with the AR system. When the camera and computer identify the shape of the marker, the virtual image of the marker overlays the real space by applying image structuring. The user can then directly observe the interaction between the marker and virtual imagery. To achieve this direct interaction, optical principles must be carefully applied to the reimage structure. The design concepts of the proposed system are explained in the following section.
2.1. The Interactive Design
The interactive structure designed in this study is based on a 3D display technique (Taiwan Patent Publication number I28830 [10]) invented by the research team. By integrating AR application concepts and the 3D display technique, the researchers could design a display system that features direct-interaction functionalities. As shown in Figure 2, the structure of the display system primarily comprises two function types, specifically, an AR identification and interaction function and a virtual-imaging function. In Figure 2, components 1 and 2 are reflective mirrors and component 3 is a Fresnel lens. Integrating these three components enables the system to render virtual images into a real-space platform. Component 4 is a virtual-image display that not only presents virtual images but also presents viewable 3D images. Component 5 is a camera that is used to extract the marker of the real space. A computer (component 6) is used to process the images captured by the imaging system (components 1, 2, and 3) and create virtual images. These images are then projected in a predefined location in which the marker can be used to interact with the real-time AR image directly. The size of the proposed structure is primarily based on two factors, namely, the focal length of the Fresnel lens and the size of the image display (component 4).
2.2. The Optical Design
In this study, component 4 was a flat display instead of a 3D device. This device produces flat images similar to those of conventional indirect-interaction AR systems. To project these images in a real space, the display position and the influence of the Fresnel lens on image quality must be considered. Figure 3 depicts a conceptual diagram of the images generated by using the interactive structure. In the figure, O represents the flat images provided by the display, I represents the generated image, M1 and M2 denote the reflective mirrors placed at 45° angles, L represents an optical condenser, and denotes the focal length. The mirror image effect is eliminated by using a double-mirror setup (M1 and M2), and inverse images are prevented because the angled placement of the mirrors is equivalent to a 180° rotation effect.
The relative positions of O and I are expressed according to the following: When the image is magnified to , the optimal image quality is obtained. Because , is observed in the system structure. When the design of the structure is symmetrical, then and . In addition, the size of I is similar to that of O. In this instance, (1) can be rewritten as where . In this study, the focal length of the optical condenser was cm; therefore, cm.
In Figure 3, L represents the optical condenser. Considering generation capacity and component availability, a Fresnel lens was used as a substitute for conventional optical condensers. However, based on diffraction theory, the quality of the final product generated by using a Fresnel lens is different compared to that of a conventional optical condenser. The reason for these differences is demonstrated by the data shown in Figure 4. In this figure, the diffraction potential of a convex lens was stratified and each layer had a thickness of (the red rectangles in Figure 4). When the red rectangles were removed, the shape of the light wavefront was unaffected and did not affect the imaging results. Thickness can be obtained by using the following reasoning: let represent the thickness of the lens medium. The optical wavelength traveling a distance of in space corresponds to a phase shift of , whereas that traveling a distance of in the lens medium corresponds to . Therefore, the difference between phase shifts is . If the reflective index of the lens medium is , then When , then .
The wavefronts of the mentioned light waves are identical. After traveling a distance of , the wavefronts of the light waves remain identical. Thus, when , the minimum thickness of the reduced lens can be expressed as follows:
However, the curved surfaces of common Fresnel lenses are imperfect unlike those presented in Figure 4, because the design and manufacturing of these lenses typically involve a second-order manufacturing process. This process sacrifices the microstructures within the lenses [11], thereby substantially reducing the thickness, weight, and size of the lens at the cost of quality degradation. This study adopted a Fresnel lens as the imaging component and used a convex lens for comparing the quality differences.
3. Testing
By integrating the mentioned structural designs and optical principles, this study developed an innovative direct-interaction AR system. Based on the operational principles presented in Figure 5, the proposed system was used to render virtual images into the real space, and the predefined marker was used to interact spontaneously with the system.
3.1. Marker Identification
To render virtual images in 3D environments and enable direct-interaction functionalities, the researchers adopted AR techniques. A camera and computer were used to complete the identification process of the marker cards (Figure 5). The marker cards used in this study were created in black and white, which was easier for the computer to identify. However, other objects could also be used in the identification process.
3.2. Interactive-Image Output
The camera captured the shapes of the marker cards, and the AR design program (VIZARD) was used to identify and confirm the shapes. In the AR program design, a predefined marker representing a 3D bouquet was used. After confirming the shape of the marker, the system produced a predefined 3D bouquet. The shape and angle of the 3D bouquet varied according to the shape and size of the marker determined by the AR system, subsequently producing images at different angles and sizes. In the AR program design, and mirrored the symmetric setting of the 3D bouquet on the liquid crystal display (LCD) and did not require adjustment because of the corresponding positions of the two reflective mirrors (Figure 2) and the two reflecting and single-diffracting processes.
3.3. Imaging
The image displayed on the LCD was rendered onto a marker by using the Fresnel lens in the imaging system. As previously explained by (1), when , the imaging of M2 is rendered at , that is, the initial position of the marker. Because component 4 is not a 3D display (Figure 2), the objects were considered flat, and thus a flat image was produced. This restricted the marker identification and display position; that is, the 3D bouquet in this test could only be displayed as a flat image.
3.4. Image Interaction
The 3D bouquet can be illustrated using the Autodesk 3ds Max software. When the image is rendered at the marker, the AR program specifies that the camera captures a specific size and determines the vertical variations of the marker, subsequently displaying the size of the marker. Based on Figure 6(a), when the distance between the marker and camera becomes great, the bouquet becomes small; conversely, at short distances, a large bouquet is displayed.
(a)
(b)
As established in the AR program, when a specific point on the marker presents a vertical line of at the central point of the camera, the 3D bouquet becomes perpendicular on the marker card. Thus, when the surface of the marker card changes line angles and , the bouquet rotates because the surface of the marker card is rotating, as shown in Figure 6(b). These results indicated that the marker card directly interacts with the predefined image.
4. Conclusions
This study successfully rendered virtual images into the real space and enabled the virtual images to interact with real-time objects. This achievement is a novel application in AR techniques. By integrating the proposed handheld 3D image-interaction structure with an image display and marker identification, AR techniques can successfully be applied to the real space, thereby enabling actual objects to be involved in real-time and direct interaction with virtual images.