Abstract
Information management includes multimedia data management, knowledge management, collaboration, and agents, all of which are supporting technologies for XML. XML technologies have an impact on multimedia databases as well as collaborative technologies and knowledge management. That is, e-commerce documents are encoded in XML and are gaining much popularity for business-to-business or business-to-consumer transactions. Recently, the internet sites, such as e-commerce sites and shopping mall sites, deal with a lot of image and multimedia information. This paper proposes an intelligent web digital image information retrieval platform, which adopts XML technology for social curation commerce environment. To support object-based content retrieval on product catalog images containing multiple objects, we describe multilevel metadata structures representing the local features, global features, and semantics of image data. To enable semantic-based and content-based retrieval on such image data, we design an XML-Schema for the proposed metadata. We also describe how to automatically transform the retrieval results into the forms suitable for the various user environments, such as web browser or mobile device, using XSLT. The proposed scheme can be utilized to enable efficient e-catalog metadata sharing between systems, and it will contribute to the improvement of the retrieval correctness and the user’s satisfaction on semantic-based web digital image information retrieval.
1. Introduction
The information services platform manages large-scale information assets compiled from heterogeneous data resources. With the information services platform, users can search for appropriate information services and effectively compose new services to satisfy their requirements and increase the value and utilization of information assets. Furthermore, the platform enhances the development of information analysis services by helping users discover correlations among large-scale and diverse data such as science data on the global environment. We harvest and analyze data provided by various research organizations, newspaper articles, and web contents describing society, and we cross-search and integrate data related to various events and phenomena. Moreover, composition of these services enables users to develop applications adaptive to changing environments, such as events and shopping in the real world [1].
Recently, the usage of web digital image information is ever increasing with the rapid development of multimedia technology. Multimedia data is commonly used on wired and wireless internet in various industrial areas including information technology area. A number of research groups are exploring different approaches to provide and transform multimedia information on various client devices, such as web browsers, iPad, and smart phones [2]. There are also various types of multimedia data in various industrial areas and they become quite essential to provide standard formats of multimedia information to allow data exchange and sharing through the Internet. One of the well-known standards for the description of multimedia information is MPEG-7. A lot of efforts have been made to represent MPEG-7 based multimedia information in XML format [3], but they are still in an early stage.
Among multimedia data, images are widely used in various applications. Retrieving images attract high and increasing interest from the wide range of applications, such as medical information systems, biological information systems, electronic museums, and e-commerce electronic catalogs. There have been a lot of research works to support content-based retrieval on images. But, we found that most of the previous research was focusing on content-based retrieval of images based on image features in global level [4].
We envision the future web as pages containing both text and semantic markup. Current web digital image information retrieval techniques are unable to exploit semantic knowledge within documents and hence cannot give precise answers to precise questions [5].
This paper is an effort to make web digital image information better utilized by adopting XML technology. To support object-based content retrieval on product catalog images containing multiple objects, we describe multilevel metadata structures representing the local features, global features, and semantics of e-catalog data. To enable semantic-based and content-based retrieval on such image data, we design an XML-Schema for the proposed metadata and show how to represent such metadata using XML documents. We also describe how to automatically transform the retrieval results into the forms suitable for the various user environments, such as web browser or mobile browser, using XSLT. The proposed scheme can be easily implemented on any commercial platforms supporting XML technology. We strongly believe that the proposed schemes can be utilized to enable efficient image metadata sharing between systems. Also, it will contribute to the improvement of the retrieval correctness and user’s satisfaction on semantic-based web digital image retrieval.
The remainder of this paper is organized as follows. Section 2 describes overview on previous image retrieval systems and XML related technologies. Section 3 explains the proposed platform architecture. In Section 4, we show a multilevel web digital image metadata modeling and represent our modeling results using XML-Schema. In Section 5, we explain the prototype implementation of intelligent web digital image metadata service platform by using a dynamic XML document search technique. Finally, Section 6 concludes and summarizes the paper.
2. Related Work
The previous image searching techniques can be classified into three branches. First approaches are simple search techniques based on keywords, which describe subjects or titles [6]. Formatted data on images and image file names are stored in databases, while images are stored in external files. Searching operations are usually performed by using descriptive keywords. Automatic extraction of image related keywords from HTML web pages is possible, but unrelated or unimportant words can be selected deteriorating retrieval performance.
Second approaches are content-based image retrieval techniques using feature vectors. Multidimensional vectors representing color, texture, and shape features of images are commonly used [7]. Image metadata are added within databases or file systems to support similarity-based searching on image data [8, 9].
Third approaches are hybrid style of above approaches. Feature vectors are extracted by image preprocessing and metadata information, such as keywords, semantic information, and visual information, which are manually or automatically added [10, 11].
2.1. Image Searching Techniques
Image metadata are added within databases or file systems to support similarity-based searching on image data [12–15]. Some systems support keyword only retrievals and others support content-based image retrievals [16, 17]. In the latter approach, image retrievals based on feature information, such as average colors, color histograms, texture patterns, and shape objects, are supported. Most of them are developed for image database applications [18, 19]. Representative examples are QBIC [20], Safe, VisualSEEK [21], Photobook [22], WBIIS, Chabot [23], and Blobworld [24]. One of the most recent research works has been done by the SIMPLIcity system which supports content-based image retrieval based on the color, texture, and shape features, while it is increasing matching correctness by utilizing local features on regions [25–27]. In the medical domain, the KMeD (Knowledge-Based Medical Database) system utilizes semantic modeling focusing on object shapes and spatial relationships between them [28, 29].
2.2. XML Techniques
XML is a standard markup language proposed for data exchanges on the web [30]. XML is proposed by W3C to describe next generation web pages [31, 32]. In database field, there have been a lot of research efforts to store, index, and retrieve XML documents in database systems [33, 34]. Some systems are stand-alone, while others are built on top of relational databases or object-relational databases [35, 36]. Another related research trend is to transform normal data stored in databases into XML documents for efficient integration of heterogeneous information resources [37]. New query languages, such as Xpath, XML-QL, XQL, Quilt, and XQuery, are proposed to support structure-based and content-based retrieval of XML documents [38–40].
One of the interesting characteristics of XML is document conversion technology. XSLT is used to transform an XML document into a document having different format [41, 42]. Figure 1 shows a situation where XML documents are transformed into HTML5 documents by using XSLT for wireless internet users.
3. Platform Design and Architecture
This paper proposes a system, called IIP (intelligent web digital image information retrieval platform), that can support intelligent image retrieval by utilizing dynamic XML documents. We utilize commercial database systems as storage systems to make our system stable and cost-effective. The system architecture is shown in Figure 2.
Data extractor preprocesses images to extract web digital image feature (global feature) and object feature (local feature). Object features are used to provide content-based retrieval from the viewpoint of component objects contained in images. Image features, object features, semantic metadata, and other meaningful digital image metadata are stored in databases. Content-based queries are submitted through IQI (intelligent query interface) and processed by feature analyzer to extract major features of query objects. The matched results are transformed into HTML5 or XML documents according to the client platforms.
The storage structures are designed by considering mapping relationships with XML documents. Our metadata structure contains global feature, local feature, semantic metadata, and major element of MPEG-7 standard. Figure 3 shows the proposed metadata schema in E-R diagram. Feature vectors for objects in digital images are stored in color, shape, texture, and spatial tables. Semantic information and other meaning metadata are stored in SemanticDS and Metadata InformationDS, respectively. One or more images can share metadata of contained objects by referencing OIDs (object IDs).
4. Web Digital Image Metadata Modeling
In this section, we propose multilevel metadata structures for intelligent digital image information retrieval.
4.1. Multilayer Web Digital Image Metadata Modeling
Web digital image metadata is data for image data. There are two types of basic metadata for images.(i)Registration metadata: image resolution (width, height), color map, compression ratio, and so forth are typical examples of registration metadata. This metadata is required to display and manipulate images. In image files, this information is usually hidden within image headers.(ii)Description metadata: digital image title, caption, keywords, natural language descriptions, and image file names are typical examples of descriptive metadata. This metadata is used to search images, when content-based retrieval operations are not supported.
Registration metadata or description metadata are not enough for content-based searching. We can further define multilayered metadata structures on top of raw image data. They are the following.(i)Global feature metadata: average values or multidimensional vectors representing color, texture, and shape of image are examples of global feature metadata. This metadata is heavily used in current content-based retrieval system.(ii)Local feature metadata: Average values or multi-dimensional vectors representing color, texture, and shape of each objects or regions belonging to a given image are examples of local feature metadata.(iii)Semantic contents metadata: subjective feelings and knowledge on images, such as concepts, meaning, category, spatial relationships, or other useful interpretations, are examples of semantic metadata.
Figure 4 explains a processing model of multilayered metadata from a web digital image (or E-catalog image). The bottom layer shows examples of objects contained in images. Global metadata layer contains global features, such as global average color, and it also represents semantic content or knowledge content of a total image. Local metadata layer represents feature vectors of each object belonging to a given image. Semantic content metadata layer represents semantics or knowledge content of each object. For example, a web digital image usually has multiple objects in an image. A feature vector assigned to each object plays an important role in image searching process. In addition, it can be utilized to improve the efficiency of searching process. There have been limitations on user’s satisfaction in existing search engines, because they mainly rely on global features only. Figure 5 shows an example of a web catalog image containing multiple objects. The first image of Figure 5 has multiple central objects “chair” and another central objects “table.” The second image has central objects “necktie” and “T-shirt.” The third image has central object “T-shirt.” But, it is easily shown that each has different color. Like this situation, central objects can have different feature vectors, even though their meanings are the same. Vice versa, they can have same feature vectors, while their meanings are different.
4.2. XML Representation of Web Digital Image Metadata
The DTD structure to represent metadata of an image is shown in Figure 6. This DTD structure is focused on the object viewpoint as well as the global image viewpoint [43]. In this design, we also adopted major elements of MPEG-7 standard. Figure 7 shows an example of XML document to describe metadata of an image containing multiple objects.
We use XML-Schema to define the form of XML documents for image metadata [44, 45]. We designed an XML-Schema to represent image metadata containing multiple objects. Figure 8 shows a metadata structure of such an image from the viewpoint of elements. Object-1 refers to all the elements, while Object-2 refers to a part of its elements. The proposed XML-Schema model allows reuse of element definition. It also allows definition of patterns or models within elements.
Figure 9 shows a document about XML-Schema in Figure 8. In this way, XML-Schema documents verify effectiveness of XML documents for multiobject images.
(a) Image object and metadata information
(b) Semantic information
(c) Feature information
Figure 10 shows the section, which defines web digital image metadata information in an XML-Schema document. It also explains a mapping scheme of image metadata part in XML-Schema document into relational databases. A user define type called “madeType” is used in order to define a child of an element in an XML-Schema document. Definition of such parent elements and child elements are mapped to table schemes in relational databases.
5. Prototype Implementation
In this section, we describe implementation aspects of intelligent web digital image retrieval platform proposed in this paper. The prototype system is implemented on the Windows Server 2012 with MS-SQL Server 2008. Web client platform is Microsoft Internet Explorer 10.0 and XML Editor Tools. We used languages, such as JQuery, JavaScript, Ajax, DOM (Document Object Model), JAVA, and JSP (Java Server Page) based on SAX (Simple API for XML). Web server is IIS (Internet Information Services) with Tomcat 7 virtual directory server that is supported by MS-SQL server.
5.1. Storing XML Documents in Relational Database System
XML document can be used to provide the platform transparency hiding the difference between systems. The proposed system provides such transparency. We keep XML documents in disk databases. Data extraction procedure from XML documents is explained in Figure 11.
To store data, XML documents are parsed and node structure of XML documents is mapped into tree structure before OpenXML function is called. The stored procedure sp_xml_preparedocument validates effectiveness of XML document. After validation, node tree handle, that can extract data from attribute and element, is returned by this stored procedure. After node tree creation is finalized, data is stored in the table to return low set data of XML document.
Procedure 1 explains the procedure to store portion of XML document describing image metadata into database tables.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
5.2. XML Document Searching
By applying XSL stylesheet into XML data, we can transmit retrieved data in different forms, such as HTML5 documents with CSS3, to web browser-based clients, such as smart phones using App. Before web browser accesses web server, a HTTP header, which contains the required pages, the machine type used by current web user, and various information about web browser, is sent to the web server. A HTTP header in HTTP request packet also contains information on user IP, OS, type of browser, type of document that can be processed in web browser, and cookie used by login information. We can submit the retrieval results to the smart device after translating them by using XSLT. For this process, HTTP header information is used to identify the type of connected machine. GetHeader method of request object is used to search the header information and “ACCEPT” attribute is used to identify the type of connected machine. To identify connected browser, “USER-AGENT” attribute is used. Figure 12 shows how image searching is done using XSLT transform process.
(a) Web digital image retrieval result by keyword
(b) Web digital image retrieval result by hybrid
6. Conclusions
XML related technologies are evolving rapidly in the Internet environments. Many application systems that utilize merits of XML technologies are recently developed. However, there have been little efforts in adopting XML techniques to realize information retrieval and efficient exchange of multimedia data.
In this paper, we presented web digital image metadata retrieval technique, which adopts XML technology. To support object-based content retrieval on product catalog images containing multiple objects, we described multilevel metadata structures which represent the local features, global features, and semantics of image data. To enable semantic-based and content-based retrieval on such image data, we designed an XML-Schema for the proposed metadata and showed how to represent such metadata using XML documents. We also described how to automatically transform the retrieval results into the forms suitable for the various user environments, such as web browser and smart device, using XSLT. The proposed scheme can be easily implemented on any commercial platforms supporting XML technology.
We are planning to develop an automatic conversion module of multimedia data such as image or video, according to the client platform of end users. There should be further researches to improve user’s satisfaction by developing various contents transformation XSLT.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.