Abstract
We discuss the importance of establishing awareness science and show the idea of using interactive evolutionary computation (IEC) as a tool for analyzing awareness mechanism and making awareness models. First, we describe the importance of human factors in computational intelligence and that IEC is one of approaches for the so-called humanized computational intelligence. Second, we show examples that IEC is used as an analysis tool for human science. As analyzing human awareness mechanism is in this kind of analyzing human characteristics and capabilities, IEC may be able to be used for this purpose. Based on this expectation, we express one idea for analyzing the awareness mechanism. This idea is to make an equivalent model of an IEC user using a learning model and find latent variables that connect inputs and outputs of the user model and that help to understand or explain the inputs-outputs relationship. Although there must be several definitions of awareness, this idea is based on one definition that awareness is to find out unknown variables that helps our understanding. If we establish a method for finding the latent variables automatically, we can realize an awareness model in computer.
1. Introduction
The number of papers using the keywords context awareness, power awareness, location awareness, and situation awareness in the SciVerse Scopus database of Elsevier is, respectively, 6,383, 1,749, 1,688, and 257 as of February 2012. Engineering interest in these areas has increased.
These engineering approaches call obtaining unknown knowledge or facts awareness. However, how do these engineering approaches differ from data mining or knowledge acquisition? Although these kind of applications are useful and important, we need other scientific approaches not only to support the engineering applications of awareness but also to extend awareness science and engineering.
One such scientific approach would be analyzing the awareness mechanisms of human beings and/or animals and constructing awareness models based on these mechanisms. Once we establish their core technologies, we may be able to make a computer with using such a model be aware of something. As the result, we can expect not only to develop data mining-like applications as has been done until now but also to progress human-machine communications, the monitoring of social networks, and new areas.
Analyzing awareness mechanisms and modeling them are the first important step to research performed in this direction. It is important to integrate ideas from and cooperate with those in ethology, psychology, mathematical modeling, engineering analytical methods, and other interdisciplinary areas. Cooperation with human sciences is especially, important though cooperation between awareness computing and human science has not been so active.
This paper has two objectives; one is to show that IEC can be an analytical tool for human sciences and the other is to discuss how we should use IEC for the analysis of awareness mechanisms and awareness modeling. For the first objective, we introduce some research using IEC as a tool for analyzing humans in Section 3 and show IEC's potential as a tool for awareness science. For the second objective, we show an application of idea in a tentative trial for further discussions in Section 4, though concrete approaches have not been proposed yet. As IEC is a core technology of this paper, we also draw a big picture to explain why IEC is necessary and introduce IEC briefly in Section 2 before these main sections.
2. Humanized Computational Intelligence and IEC
In this section, we would like to emphasize how exploiting human factors has become increasingly important in computational intelligence. We will refer to the integration of human factors into computational intelligence as Humanized Computational Intelligence. An engineering solution is designed to optimize some task, and conventionally the engineers would create the design based on their knowledge, experience, and even preferences. This design approach shares some similarity with the approaches taken by artists or craftspeople. We may call this kind of design, based on the engineer's capabilities, a first-generation engineering design. In the second generation of engineering designs, the human engineer is replaced with optimization techniques or computational intelligence algorithms using autodesign methods. To clarify, computer-aided design (CAD) is not an autodesign method but rather a tool used by engineers to input their designs into a computer; designs using CAD belong to the first-generation. In the third generation of engineering designs, there exists cooperation between human engineers and autodesign methods. If perfect autodesign methods existed, the second generation would be the optimal. Unfortunately, no such methods exist. Since autodesign methods and human experts have different strong points, the best choice is to combine them and blend their strengths. Figure 1 illustrates some examples of this third generation of design. In conventional robotics and control applications targeting the natural environment, designs get information from sensors and make decisions regarding actions. Recently, consumer robotics have achieved considerable success and emotional reactions such as cute, friendly, or safety have become the most important factors in determining the sales for this kind of product. Conventional approaches cannot design for such emotional reactions without a human’s subjective evaluations during the design process. Data mining acquires knowledge from a database, and practical techniques in this area have been established. However, a computer cannot evaluate how the obtained knowledge is important and therefore cannot obtain qualified knowledge without the evaluations of domain experts. Filters for sound or image can be implemented by mathematical signal processing process algorithms on the computer. Although specifications, such as signal-to-noise ratio or error between processed signals and target signals, can be used to design filters for a given target and optimization techniques can be used during the design, really, the best processed signals are frequently decided based on human vision or auditory inspection rather than the numerical specifications alone.
These examples show how human factors have become important in engineering design and how computational intelligence would benefit from the embedding of these human factors. That is, we need Humanized Computational Intelligence. Historically speaking, engineering approaches in artificial intelligence have handled only human logic or knowledge, while human beings have two aspects: the logic or knowledge aspect handling reasoning, knowledge expression, knowledge acquisition, associative learning, associative memory, and so on and subjective aspect handling intuition, preference, subjective evaluation, perception, cognition, and so on. The Humanized Computational Intelligence should deal with the latter.
Interactive Evolutionary Computation (IEC) is a tool well suited for realizing Humanized Computational Intelligence. IEC is an optimization based on an IEC user's subjective evaluations instead of a fitness function or measured fitness (see Figure 2). There are many tasks that are difficult or impossible to evaluate quantitatively; for example, there is no way to measure how sounds from a hearing aid are good to its user, but we must optimize its fitting even if we have to fit it based on a trial-and-error strategy. IEC can solve these kinds of optimization tasks.
Since the first IEC application by Dawkins in 1986 [1], many IEC papers have been presented. IEC research is roughly categorized into two: finding new IEC applications and reducing IEC user fatigue. IEC applications can be roughly categorized into three areas: artistic applications, engineering applications, and others. These applications include artistic applications such as graphics, music, industrial design, and facial design, engineering applications such as acoustics, image signal processing, data mining, robotics, control, and other applications such as geology, education and games. For more details, see the IEC tutorial and big IEC survey in [2].
Recently, new types of IEC applications have been proposed though most IEC applications are still optimizations of target systems. A new approach in IEC research is to use IEC as a tool for analyzing humans. As mentioned, evolutionary computation optimizes a target system based on human evaluations in an IEC system. We may understand the human's evaluation metrics or mechanisms by analyzing the target system optimized by the human evaluation. This approach has a similarity to reverse engineering which estimates inputs from outputs. We describe this approach in Section 3, and the approach is the background to the main idea presented in this paper described in Section 4.
3. IEC for Human Science
3.1. Measuring Emotional Dynamic Range of Human Mind
Some therapists have proposed, through their experiences, that the range of emotional expressions in the faces of schizophrenics is smaller than those of mentally healthy persons. There has been, however, no way to confirm this empirically. This was a motivation for measuring them indirectly using IEC. Although it is difficult to measure the dynamic emotion ranges from facial images, it is possible to measure patients' decision makings for emotions. Here, we can use IEC.
Schizophrenics patients and mentally healthy university students used IEC to create happy and sad lighting impressions based on their subjective degrees of the happy and sad impressions. We conducted a human subjective test and 33 subjects compared created lighting impressions by schizophrenics users and those by mentally healthy users, and we applied a statistical test to the human subjective test results (Figure 3). If there is no significant difference between two user groups, we may say that their dynamic ranges of happy-sad are almost the same. We measure emotional dynamic ranges from this relative comparison.
An IEC-based 3-dimensional computer graphics (3D CG) lighting design support system [3, 4] was used for this experiment. This system is used to design CG impressions matching to the given design concept by optimizing the coordinates of the lights in a 3D space, on/off of the lights, lighting strength, and types of light sources. Lighting colors are optimized, too, for color lighting design.
A genetic algorithm (GA) was used as an EC and generated nine 3D CG images. The IEC user gave fitness values in five levels for each image, and GA optimized the mentioned parameters based on the fitness. IEC users consist of three schizophrenics and five mentally healthy subjects. The experimental system is shown in Figure 4.
After obtaining 3 + 5 CG images per design concept, 33 subjects used a five-level rating scale to compare 28 () pairs of each of happy and sad impressions using Scheffé's method of paired comparison that is a statistical method, analysis of variance (ANOVA), from paired comparison data.
The obtained psychological scales are shown in Figure 5. Psychological yardsticks for a happy constructed scale are 0.19 and 0.16 for () and (), respectively, and those for a sad constructed scale are 0.24 and 0.21 for () and (), respectively. These experimental results show statistical significances as PT, NH, PT < PM < NY < NK, and NN < NS on a happy constructed scale and NY, PK, NH < NK, NS < NN, PM, and PT on a sad constructed scale.
(a) happy case
(b) sad case
This approach illustrated IEC's applicability through its concrete realization in an experiment to measure the dynamic range of emotional expression capability in schizophrenics and mentally healthy subjects. There has been no technique for measuring this kind of mental dynamic range so far, but IEC has the potential to be the technique. This point is the foundation for our expectation that IEC can be used as a tool for analyzing human awareness mechanisms.
IEC also has a potential for psychiatry. So far, schizophrenics’ symptoms and daily-life functions have been diagnosed and measured using PANSS [5], LASMI, and other check lists of patient actions. These experimental results demonstrate the potential of IEC to be a new diagnostic tool in addition to or instead of these check lists.
3.2. Finding Unknown Auditory Facts
It is impossible to measure how users of hearing aids or cochlear implants really hear sounds, but they can report whether the sound is good or bad for them. This presents a typically adequate scenario equivalent to IEC application, and IEC was applied to hearing aid fitting in [6] and cochlear implant fitting [7]. Since the IEC-based fitting is quite different from conventional fitting approaches, we were able to obtain new audio-psychophysiological facts from these new approaches.
There are two hypotheses for conventional cochlear implants fitting. One is “the more electric channels are, the better” to increase frequency resolution. Electric channels are set along the basilar membrane, which is equivalent to be set along the frequency axis. This is why the interval between neighbor electric channels corresponds to the minimum frequency difference that an audio-nerve system can distinguishes. Another is “the bigger dynamic range of electric stimulus, the better” to hear sounds from the minimum intensity level (threshold value: T-value) to the maximum one (comfort value: C-value). These two hypotheses look natural, and all existing current cochlear implants fittings are based on them.
A French team applied IEC to cochlear implant fitting and obtained quite strange results in [7], as shown in Figure 6. The dotted lines show C-values and T-values for the 15 electric channels obtained by a conventional fitting method, and the C–T ranges are set to be maximized. However, the fitting characteristics obtained by IEC-based fitting method are quite different from those of a conventional method as shown by solid lines.
Only 3 or 4 among 15 electric channels work and their dynamic ranges are quite a bit narrower than their C–T ranges. In spite of such poor fitting characteristics from the point of view of a conventional fitting, the IEC-based fitting characteristics showed higher word recognition rate; whereas a rate of around 50% was obtained with conventional fitting, more than 90% was achieved with the IEC fitting. This was a surprising result. The two hypotheses for cochlear implant fitting cannot explain these experimental results.
This fact implies that there must be unknown audio-psychophysiological facts. As it was found by IEC, IEC may be able to find other unknown facts in several areas of human science as an analysis tool. This is why we can expect to use IEC as a tool for analyzing awareness mechanisms.
We applied IEC to hearing aid fitting and showed that the IEC-based fitting realized almost equivalent performance to the fitting done by a fitter of the top level [6]. After this engineering approach, we analyzed the characteristics of the fitted hearing aids in different sound environments.
Some facts that we found thanks to the IEC approach are: (1) loudness characteristics obtained by conventional fitting method using band noises or pure tones differed from those obtained by IEC fitting using voices (2) differences among fitting characteristics obtained by IEC fitting using several different speeches with/without noise are small, while those are quite different from fitting characteristics obtained by IEC fitting using music with/without noise. These facts were not known until an IEC fitting method was applied.
3.3. IEC with Physiological Responses
IEC is an optimization system wherein an IEC user evaluates the outputs from a target system based on a priori knowledge, experiences, and/or preferences and makes the EC optimize the target system. Since the user evaluation is a subjective evaluation, we may say that IEC is a system applying psychological feedback to the EC.
Extended IEC is another IEC framework that extends the feedback from psychological data of an IEC user to his/her physiological data [8]. Let's explain it using the example in Figure 7. Let the goal or ideal physiological responses of relaxation or excitement, for example, be supervised data. The physiological responses of an IEC user who is watching movies are measured, and the EC optimizes the coefficients of an image filter to minimize the differences between the supervised data and the measured physiological responses. Thanks to this framework, we can extend IEC to handle human physiological data.
Extended IEC is an optimization framework rather than an analytical tool mentioned in Sections 3.1 and 3.2. However, note that we realized the IEC's potential as a tool for analyzing human awareness mechanism by changing our view from the IEC as an optimization tool to the IEC having psychological feedback as an analytic tool for human science. In the same way, we would like to expect that IEC becomes a tool for awareness science by extending it to the Extended IEC.
4. IEC Approaches for Modeling Human Awareness Mechanism
4.1. Analysis of Human Awareness Mechanism
Psychological awareness is composed of three layers: the sensory layer, the perceptual layer, and the cognitive layer. In the case of sound, sound loudness or frequency is handled at the sensory layer; melody or word voice is distinguished and recognized at the perceptual layer; meaning of recognized sounds is handled at the cognitive layer. Accordingly, we can open a road for an ambulance when we hear its emergent siren.
Besides these three psychological layers, humans have a KANSEI layer that handles values of input stimuli, which are not handled by psychology. Although there are several definitions of this concept in the areas of information science, psychology, linguistics, design, and others, we define it as the capability or function of handling subjective values of inputs to a human being in this paper. For example, “I like this relaxed melody” is a process resulting from this KANSEI layer. There are many IEC applications using this layer, and we take account of the function in the layer in our analysis of the awareness mechanism.
We have different levels of awareness in different layers, as shown in Figure 8. As an example with the KANSEI layer, suppose you come to be attracted by a pot. The preferable feeling comes first, but it may be hard for you to immediately explain its reason. After a while, you may become aware of the reason, for example, the balance of color and its shape or its similarity to a toy that you played well with in your childhood, and thus you become able to explain the reason for your preference.
We should consider the mentioned psychological layers in our analysis of the awareness mechanism and the construction of awareness models.
One of the methods we can use to analyze the awareness mechanism would be to construct an input-output relationship for humans. Continuing with the above-mentioned example wherein a pot is evaluated, anyone can answer how they like the pot (), when they see it. However, it is quite difficult to explain the relationship between a visual image of the pot (input ) and the evaluation (output ). After thinking for a while, they may become aware that the vertical-horizontal ratio of the pot () and the curvature of design pattern () are the points underpinning their evaluation and become able to explain the reason for their preference.
In other words, we may say that they are aware of the two hidden variables of the vertical-horizontal ration of the pot and the curvature of design pattern. That is, we can say that “ in was so complex that the subject could not explain the input-output relationship at first glance. After that, they found the latent variables and and could interpret them as and thus explained their evaluation, , using a simple relationship involving and .”
4.2. Modeling of the Human Awareness Mechanism
We describe how we are proposed to realize the awareness mechanism in computer. We have not reached the stage of reporting our experimental results yet, but are rather at the stage of discussing our idea.
Step 1. The first step is to make an evaluation model of an IEC user. The IEC user inputs graphics, movies, sounds, and others from a target system and outputs his or her evaluations of them. We can make an IEC user model using a model system, a machine learning algorithm optimizing the system, and his/her inputs and outputs as training data (see Figure 9). The learning algorithm optimizes the model system using the training data. The model system includes neural networks (NN), fuzzy or crisp rule-based systems, neurofuzzy systems, and others; the machine learning algorithm includes a learning function of NN, and evolutionary computation.
The obtained user model can be analyzed and used to create an awareness model. When the relationship between the inputs to and outputs from the obtained evaluation model of an IEC user is simple, a computer may be able to easily explain why the IEC user evaluates the outputs from a target system from the obtained rule-based systems or neurofuzzy systems. The obtained rules are themselves the explanations. However, when the input-output relationship is complex, we need to turn to a second step, which we will now describe.
Step 2. Resolving a complex system into several simpler systems is one of methods for decreasing complexity. We assume here that there are hidden variables or latent variables between the inputs and outputs as shown in Figure 10. Then, we can interpret that analyzing an awareness mechanism is to find the latent variables.
For example, when we imagine the type of students from their examination scores in mathematics, language, and physics, and from their heights and weights, maybe we can obtain an impression of the students with regards to their scholastic ability or their body builds.
Once a computer can extract the unknown latent variables between inputs to an IEC user, that is, outputs from a target system, and outputs from the IEC user, that is, his/her subjective evaluations, the computer may be able to come to explain why the IEC user evaluates the outputs from the target system with the subjective evaluations using the extracted latent variables. This is one example approach for constructing awareness mechanism.
The key point is how we can obtain such latent variables. Possible methods include (a) structure analysis of a structured NN-FS model, (b) introducing statistical methods for finding latent variables, (c) making a learning model by a math equation using genetic programming and analyzing the obtained equation, and (d) others.
When the relationship between inputs and outputs to/from an IEC user is complex, for example, it is strongly non-linear, it must be difficult to find latent variables unlike the mentioned scholastic ability and body builds in the above. As the first stage of constructing awareness model, we should start the simple case using the discussed approach and then develop new method for finding latent variables.
5. Conclusions
We emphasize the applicability of IEC for awareness science, especially for analyzing the human mechanism of awareness, by showing its applicability for human science with some concrete approaches. At this time, we are at the start line in the process of using the proposed approach and must develop concrete routes towards the goal of analyzing the awareness mechanism and providing an awareness machine model for engineering applications. We hope that introducing the applicability of EC in this paper increases the interests of researchers in awareness computing into not only engineering aspects of the awareness computing but also its scientific aspects and helps them in their research.
Acknowledgments
This paper was made based on the author’s keynote speech at the Second International Symposium on Aware Computing (ISAC 2010) held in Tainan, Taiwan. The author would like to thank Prof. Qiangfu Zhao of the University of Aizu and other committee members for giving the author the opportunity to present his views on approaches for awareness science at ISAC 2010. This work was supported in part by Grant-in-Aid for Scientific Research (23500279).