Abstract
This paper presents results from an empirical study conducted with a subliminal teaching technique aimed at enhancing learner's performance in Intelligent Systems through the use of physiological sensors. This technique uses carefully designed subliminal cues (positive) and miscues (negative) and projects them under the learner's perceptual visual threshold. A positive cue, called answer cue, is a hint aiming to enhance the learner's inductive reasoning abilities and projected in a way to help them figure out the solution faster but more importantly better. A negative cue, called miscue, is also used and aims at obviously at the opposite (distract the learner or lead them to the wrong conclusion). The latest obtained results showed that only subliminal cues, not miscues, could significantly increase learner performance and intuition in a logic-based problem-solving task. Nonintrusive physiological sensors (EEG for recording brainwaves, blood volume pressure to compute heart rate and skin response to record skin conductivity) were used to record affective and cerebral responses throughout the experiment. The descriptive analysis, combined with the physiological data, provides compelling evidence for the positive impact of answer cues on reasoning and intuitive decision making in a logic-based problem-solving paradigm.
1. Introduction
The use of technology to build Intelligent User Interfaces (IUI) has revolutionized the way computers interact with human beings. Examples of these IUIs can be found in virtually every aspect of our lives, such as mobile media sharing [1, 2], intelligent vehicular assistance [3], and mental spelling machines [4]. One growing area of research within the HCI community recently has been focusing towards the design of smart interfaces for learning. Indeed, a smart interface should be able, in a learning context, to detect the user’s emotional and cognitive states in order to adjust and adapt the teaching material accordingly. Such adaptive systems relying on efficient IUIs are known as intelligent tutoring systems and are comprised of multiple goal-specific modules to aid the learner. One of these modules, called the tutor, makes use of cognitive pedagogical strategies and affective states to properly model human cognition according to the learner’s constant needs and evolution. The tutor takes advantage of recent development in IUI to efficiently communicate with the learner. This multidisciplinary field of research tries to accomplish this daunting goal of user-modeling and adaptation by implementing the most recent evolutions and advances in various research areas such as artificial intelligence, neuroscience, and cognitive science. One of those recent advances this past decade has been the shift in emphasis for cognitive science from learner’s performance to learner’s experience during learning. Indeed, affective states, motivation, and knowledge construction have been extensively measured and explored [5–8] and have shown that emotions are an important component in skill acquisition and play a pivotal role in learning. Indeed, researches in neurosciences and cognitive psychology have shown that emotions are widely related to diverse cognitive processes, such as attention, problem solving, and decision-making [9, 10]. Emotions influence our behavior and play an important role in our every-day decision-making processes [11]. Cognitive activity is also fundamentally related to emotions [12]. Cognitive process such as problem solving and decision making not only depend but are greatly intertwined with the individual’s emotional state [13]. Moreover, emotions are essential actors for creative thinking, inspiration, as well as concentration and motivation [10, 14]. It then becomes vital for an HCI system to detect and recognize these emotional and cognitive states, via physiological sensors or otherwise, and relay them to the pertinent ITS modules. Hence, learning systems would be able to intelligently adapt their communication and interaction with learners through adaptive HCI systems.
Nevertheless, a major component of learning and decision making when solving problems has been mostly neglected in this research field: human unconscious cognition. Indeed, the cognitive unconscious as a major player is the integration and interpretation of complex material with regards to decision making and possibly learning. To that end, a large body of work in neuroscience and other fields has put forth compelling evidence that learning simple-to-complex information can be done without perception or complete awareness to the task at hand [15–18]. The idea that any information projected too fast to be consciously perceived by a learner (called subliminal projection) has been the focus of much research in neuroscience and cognitive psychology. A variety of IUIs have been designed for such a purpose, ranging from simple two-dimensional interfaces (2D) to complex 3D immersive ones [19–23]. Furthermore, the recording of the neural electrical activity, called EEG activity, present in the brain during the occurrence of these mental processes is also an active research area in HCI. Indeed, an IUI that detects brain activity can recognize and quantify the user’s engagement level in specific activities [24]. However, in the HCI community, EEG activity is mainly used to construct brain computer interfaces mainly aimed at character spelling or item recognition [4, 18, 25–27].
Unfortunately, very scarce research in the HCI community has employed affect, subliminal priming and EEG for educational purposes. Indeed, the existence of perceptual learning without perception has not only been proven, but also replicated in one study we did two years ago [19]. In this study, we presented a novel subliminal priming technique built into an adaptive IUI aimed at boosting performance by enhancing the learner’s deductive reasoning capabilities. Our findings concur with the literature: subliminal stimuli, which is stimuli projected below the threshold of conscious perception, can enhance learning and increase the deductive reasoning of learners only if carefully and intelligently constructed before being integrated into an IUI. However, this initial study suffered from a major limitation: the exclusive use of positive subliminal priming (cues designed to help learning, not hinder or interfere with it). Thus, we did not know what effect might negative cues, or miscues, have on learning and performance when integrated into our IUI system.
In the present paper, we intend to follow up on our recent study by designing an evolved version of our adaptive IUI with our novel subliminal learning technique aimed at enhancing the learner’s inductive learning capabilities. This new IUI implemented in our tutoring system will marry positive and negative primes as well as affective sensors in the stringent 2D environment resembling online tests. Affective sensors will serve as important indicator of the learner’s emotional and cerebral state when faced with the decision of correctly answering a question. When facing a question, learners can either (a) guess the answer or (b) correctly infer the solution (since this is an exam and the solution is not presented to them). We believe that IUIs would be able to detect both outcomes ((a) and (b)) by recording and analyzing emotional and cerebral data. We also believe, based on the extensive literature to that effect, that subliminal priming should influence both outcomes. We thus intend to investigate the relevance of augmenting our IUI with cerebral sensors and using our subliminal teaching technique by stating two HCI-research questions. First, does integrating different subliminal cuing types (positive, control, and miscue) into this newly designed IUI enhance or hamper the learner’s inductive reasoning capabilities and performance? Second, what significant physiological impacts can this newly designed interface with our subliminal teaching technique have on learner’s performance during question answering as well as on the entire learning process?
The organization of this paper is as follows: in the first section, we present previous work related to various aspects of our research. The second section will discuss priming in HCI, more specifically from an educational standpoint. The third section will lay the bases of physiological recordings and importance to education and learning in general. The fourth section describes the experiment setup and depicts the various aspects related to subliminal cues and miscues. The fifth section presents the obtained results which are discussed in section six leading to the last section where we conclude and present future work.
2. Related Work
The present work employs subliminal priming, affect, and EEG to investigate the possible contribution of priming to enhance learning and more specifically inductive reasoning. To the best of our knowledge, there are only two similar papers in the ITS/AIED (Artificial intelligence in education)/HCI community that employ subliminal priming and EEG in an educational context [28, 29]. The first is our own work where we presented cerebral evidence, namely, a P300 component (see background on EEG and affective data below), to confirm that both cues and miscues were interpreted in the brain, but with slightly different levels and amplitudes. The second, however, greatly differs from ours in two ways: (1) the priming used is of a personal and not of a cognitive nature (self-words were projected on the screen instead of cognitive cues and miscues) and (2) the authors’ main objective was to use priming to enhance learner’s self-esteem first and foremost. Thus, any observed performance increase could be considered as a probable side effect of the higher level of self-esteem reported. In the present work, cues and miscues were constructed precisely to enhance performance by augmenting the learner’s inductive reasoning abilities. However, we will briefly describe in this section the work done in other related fields that have been the driving force behind the inspirations and motivations for this work.
An existing dissociation between performance and consciousness has been known to neurologists for over 100 years with a disorder called blindsight. Patients in this condition are unable to indentify anything presented in one side of their visual field even though their eyes (retinas) are perfectly healthy. Nevertheless, Poppel showed that patients with blindsight can unconsciously identify and process simple visual information presented to their “blind” side [30]. Furthermore, DeGerder demonstrated that blindsight patients can recognize and identify complex visual stimuli (emotional faces) when such stimuli is presented subliminally [31]. These results, and many others, have been reproduced successfully on healthy patients and have contributed to clarify, and more importantly, to justify the anatomical separation that exists between conscious and unconscious cognition [32]. Unfortunately, no work in the HCI community, besides ours [19, 33], has attempted to use affective and cerebral data in order to assess the impact of subliminal cues and miscues on the learner’s performance, deductive and inductive reasoning, as well as decision making in a problem-solving context. However, a handful of papers in various fields have claimed the use of subliminal priming as a support for memory in the HCI community.
One of the early works regarding the implementation of subliminal cues for task-supported operation within a software was the text editor program of Wallace [23]. In this experiment, Wallace et al. investigated the response of subjects when requiring help from a text editor. They found that the frequency at which subjects demanded help was much lower when the required information was presented in subliminal matter. Another very important finding is the fact that all previous studies did not have the intended results because the projected stimulus did not take into account the specifications of the computer such as screen resolution and refresh rate. The memory glasses by DeVaul et al. [20] used wearable glasses that projects subliminal cues as a strategy for just-in-time memory support. The objective was to investigate the effect of various subliminal cues (correct and misleading) on retention in a word-face learning paradigm and compare recall performance. Another use of priming for memory support can be found in the thesis of Shutte [34], where the author assessed the effects of brief subliminal primes on memory retention during an interference task. The thesis examined in detail multiple parameters of subliminal projections such as duration, relevance, and contract. Although the results of these priming seemed very encouraging, the author cautions HCI designers that misusing subliminal priming that can lead to critical disruptions of ongoing tasks. Further work from Mitchell et al. [35] has hypothesized that recognition performance in a standard item-based forgetting paradigm may be altered by subliminal cues. These authors believe that these cues can automatically activate different mnemonic strategies, thus enhancing memory retention and recall. The results have shown that item-based words primed with the word “remember” seem to be better retained than “forgot” by a very slim difference, however. Besides seeming to impact memory, subliminal priming can also have an emotional consequence on learners. Very recent work from [21] put forward an interesting effect that subliminal priming can have on the self-attribution of authorship of events. Subjects were asked to compete against a computer in removing nonwords such as “gewxs” from a computer screen in the fastest time possible. After a determined amount of time, the computer would remove the word. Subliminal primes of self-associated words like “I” and “me” before an action increased the personal feeling that it was the participant that eliminated the non-word and not the computer, thus increasing the feeling of self-authorship of events.
Furthermore, monitoring the emotional reactions of the learner using physiological sensors in a learning environment using IUI is quite explored as well. Indeed, the empathic companion is a good example where multiple physiological sensors, namely, galvanic skin response, heart rate, and respiration were taken in real time to analyze and adapt the tutor to the emotional reactions of the learner in a virtual 3D ITS [36]. Bosma and André [37] have merged two techniques in order to resolve ambiguities in dialogue acts. They used a Bayesian network approach to determine an emotional state and a finite state machine method to study the meaning of dialogue acts according to physiological data. Further research has analyzed a more detailed and relevant emotional significance of physiological signals, either in complex learning or gaming [38–41]. There is unfortunately very scarce work to our knowledge in the HCI field where EEG and emotional data are used to enhance or adapt learning according to user’s demands besides two notable exceptions: (1) Stevens et al. employed a light-weight EEG monitoring system to evaluate changes in cognitive workload, engagement and distraction as students acquire problem solving skills [42], and (2) Chaouachi et al. have started to investigate the important link that may exist between affect and mental workload as measured by a well-known EEG mental engagement index [24].
The current paper is presented through a newly designed IUI aimed at enhancing the learner’s inductive reasoning. The new IUI is interplay between subliminal priming, EEG, and affective data. Thus, a primer on subliminal priming in HCI will first be presented, followed by a background survey of EEG and affective data, since these sensors play an important, and active, part in our study.
3. Subliminal Priming in HCI
As mentioned before, this research focuses on subliminal stimuli to enhance learning in a tutoring system. Before going further, we need to clearly establish the terminology that will be used in this paper. Indeed, the simple mention of the word subliminal can lead to confusion. We first state that a learner is consciously perceiving a stimulus if they are fully aware and able to identify the given stimulus. In the example of a computer screen saver where one would see a flying white box over a black computer screen, the fact of identifying the white box is sufficient to state that the learner consciously perceives it. However, if we project very quickly, 10 ms for example, the letter “A” in Times New Roman 10 points in light gray in the bottom left corner of the screen, then we can state that they have consciously perceived the box but not the letter. This is due to the neurological properties of conscious perception: the existence of a threshold for conscious access [15]. It is precisely this threshold that we accept as being the line between conscious and subconscious perception (that some can call unconscious as well). The technique used to project a given stimulus below this threshold of awareness is called subliminal priming as opposed to a liminal priming where the stimulus can be consciously perceived above the threshold.
Moreover, a subliminal stimulus must, therefore, be presented for a very short time and can vary in intensity, duration, frequency, and shape. It has, however been established that subliminal stimuli can bias behavior and even yield better results than no subliminal stimuli under specific experimental conditions [21, 43]. Indeed, simply sending subliminal messages without caring for context nor desired goals and emotions can prove to be inefficient or simply useless [17, 34]. The cognitive process that takes place when the stimulus (vocal or visual) is either unattended or subliminally projected is a stage that we call subliminal processing. Contrary to popular belief, high-level semantic processing has been observed during this stage [44]. One interesting example is an early study by [45], where subjects were given unattended stimulus in the form of words and were capable of discriminating between their alternative meanings. Another fascinating experiment placed forward the long-term effect of subliminal priming on the visual system [43]. In this experiment, subjects learned a task without actually realizing it. Participants were asked to focus their attention on random letters appearing on a computer screen while at the same time imperceptible moving dots in different directions were appearing just outside of their visual field. After a month’s training, the participants subconsciously learned to recognize and better identify the dots movements even if their attention was concentrated on the letters appearing in their visual field on the screen.
Furthermore, the impact of subliminal stimuli on behavior has been proven to be reliable only if the stimulus is goal relevant and people are motivated (either intrinsically or otherwise) to pursue the goal [17, 46]. Indeed, a few stunning examples of such impact on behavior is presented by Karremans et al. [47] where they argue that for such subliminal primes to be processed (say verbal), the primes have to consist of one or two short words and not of whole sentences. The notion of goal-relevance as well as proper cue construction is thus crucial for the success of the hoped effect.
The most relevant work, however, regarding our research has been done by Lowery et al. who demonstrated that subliminal primes can increase performance on midterm exams compared to neutral primes and that subliminal priming may have long-term effects on real-world behavior [48]. Thus, we believe possible, with carefully designed subliminal stimuli, to enhance learning without conscious perception by the learner. We also believe that this new form of learning can trigger a set of positive emotional conditions in the learner, such as motivation and engagement. In turn, these positive emotions can enhance learning, inductive problem solving, and possibly decision making. To that end, we designed our cues to be projected at strategic spatial positions on the screen by following the directions given in a theoretical taxonomy that differentiates between conscious, preconscious, and subliminal processing proposed by Dehane et al. [32]. This taxonomy is based on the global neural workspace hypothesis [49] that distinguishes prime accessibility in the brain based on stimulus strength and top-down attention. Furthermore, this taxonomy has been quite recently empirically validated by Van den Bussche [50].
4. Background on EEG and Affective Data
This section will present background information on the importance of measuring physiological data both in terms of EEG in the brain as affective data in terms of valence and arousal, two important emotional dimensions used in this research.
4.1. Background on EEG
It has been known, as early as 1929, that the on-going electrical firing activity of groups of neurons can be monitored with probes, referred to as electrodes, placed on the scalp, outside the brain [51]. The recording of such small activity (in microvolts, 10–6 volts) is called electroencephalography or EEG and is typically done with a polygraph displaying the continuous changes in voltage over time. There are roughly two main paradigms when working with EEG raw data (see Figure 1).
(a)
(b)
(c)
The first paradigm, time/frequency analysis (typically using a fast Fourrier transformation or FFT), is an overall look at specific frequencies over a period of time. This technique is mainly used to clinically help diagnose multiple disorders such as epilepsy, sleep disturbances, and attention disorders by employing neurofeedback protocols [52]. This type of analysis gives relatively fair information about the overall mental state of an individual based on brain locations being surveyed. For example, Alpha waves, or bands, are 8 to 12 Hz relatively large amplitude EEG waves that are usually associated with a relaxed but awake state [52]. However, Alpha asymmetry patterns between left hemisphere (LH) and right hemisphere (RH) recordings while adult subjects are closing their eyes (called eyes closed) are common patterns in depression (C3 LH Alpha > C4 RH Alpha). Thus, specific mental changes can be assessed by comparing recordings from multiple scalp sites using a 3D spectrogram such as the one depicted in Figure 1. Although no consensus is presently available regarding band distribution, the following ranges are the most commonly agreed upon by therapists: Delta (1–4 Hz), Theta (4–8 Hz), Alpha (8–12 Hz), Beta (12–32 Hz), and Gamma (38–42 Hz) [52].
Indeed, in an educational setting, attention and semantic memory performance can both be ascertained from variation of Alpha and Theta bands [52, 53]. Changes in the Alpha band has also been known to correlate to arithmetic strategy used in addition and subtraction which is very close to what we are trying to improve in our study [54]. Regarding decision making, Alpha, Beta, and Delta have been shown to be valuable markers for the balance between reward and punishment [55]. Indeed, slow Delta and Theta waves have been linked to subcortical brain regions involved in affective processes [56, 57] whereas fast Beta activity has been associated with cognitive control processes [58]. Also, specific bands, such as the Gamma and 40 Hz bands, have been quite extensively studies when it comes to insightful problem solving [59, 60]. Another exploration made in the 1980s was the exploration of asymmetries between brain regions regarding specific cognitive functions [61, 62]. These findings yielded some interesting metrics, called ratios, to measure and express mental operations. Thus, recent work in asymmetry analysis relevant to our research has revealed higher Beta asymmetries from the left brain during complex arithmetic processes compared to simple computations [63]. This area of research can potentially be quite invaluable in accessing whether the learner is engaged in far too much mental arithmetic than required for a specific lesson.
The second paradigm, however, called ERP waveform analysis, takes advantage of neurological properties of the raw EEG signal to gain better insight into the cognitive processing mechanisms in the brain. It does so by synchronizing (also called locking) the recording of the raw EEG to an event. By event we mean an action such as viewing a picture from the internet or displaying emotional versus nonemotional words on a computer screen (e.g., love, murder versus chair, and table). The resulting EEG activity is called an “Event-Related Potential” (ERP) which can be easily distinguished from the raw EEG (top part of Figure 1) by its consistent morphological shape (e.g., the P300 bold line of Figure 1 appearing approximately 300 ms following an event). While the raw EEG is made up of all brain activity at a particular point in time, specifically left and right hemisphere in this example, the ERP is only part of that raw activity associated with the processing of a specific event. ERPs are defined by their amplitude and latency (the time taken to attain peak amplitude). To visualize ERPs, one must use signal processing techniques to eliminate nonevent activity by using specific band-pass filters and averaging multiple epochs together [64]. The averaging process tends to decrease the influence of random activity (e.g., the gray dotted lines in Figure 1) while maintaining the consistent event-related activity (e.g., the solid line in Figure 1). By convention ERP researchers separate ERP waveforms into several basic parts called components [64]. Cognitive scientists have been most interested in the long-latency ERP components which include the N400 and P300 components. The letters P and N represent the polarity of the component, and the numbers represent their latency after stimulus onset (e.g., the P300 is a positive component peaking at 300 ms). In general, components occurring after 250 milliseconds, namely, the P300 component, are thought to reflect higher-order cognitive processes (e.g., selective attention, resource allocation, and language) [25, 64].
In the light of what has just been presented, it is important to mention that one of the main goals of our research in general, and not just this presented work, is to endow an IUI with the capability to recognize a specific mental activity related to the learner’s status and inform the ITS who would react accordingly. For example, suppose that a learner is trying too hard to focus on complex mathematical operations when they should be looking at the “whole picture”. An interface not equipped with EEG sensors would have to rely on imperfect data, such as previous response time to questions current time spent on lessons or worse, disturbs the learner to establish a self-report for a proper diagnostic. This can easily hinder the learning session, especially if the intervention is based on “approximate” and incomplete data. In the case of an EEG-evolved IUI, the problem is far less complicated because Beta asymmetries from the left brain would immediately tell and inform the ITS of the present mental state of the learner. A strong dominant presence of Gamma and 40 Hz activity for example during a learning session is strongly indicative of intuitive decision making and learning. Relying on EEG as well as standard data will hopefully enable us, in future work, to directly classify the mental state of the learner in a precise manner and thus adapt learning accordingly.
4.2. Background on Affective Data
The literature regarding the ITS and HCI commonly denotes the use of the following physiological signals: skin temperature (ST), respiration (RESP), mainly heart rate (HR) and blood volume pressure, (BVP) for cardiovascular activity, galvanic skin response (GSR), and surface electromyography (SEMG). ST changes when the body undergoes a stress response. In an unconscious process, blood then flows away from the extremities such as hands resulting in a decrease in ST, thus colder hands [52]. RESP is accepted as being an automatic response that increases when provoked by physical exercise or by the expression of thoughts resulting in an emotional reaction, namely, fear and sadness [65]. Respiration amplitude and frequency can be important indicators for the presence of anxiety [66]. GSR is a measure reflecting the electrical resistance of the skin. A high GSR signal is often correlated with a high stress level [67]. GSR has a direct link with arousal [68] and is often used to reflect emotional reactions as well as cognitive activities [52]. These sensors are known to reliably measure specific emotional activations and are widely used for emotional detection. Indeed, as emotions can be characterized in terms of judged valence (pleasant or unpleasant) and arousal (calm or aroused), collected physiological signals are analyzed according to the arousal/dominance emotional space. GSR increases linearly with a person’s level of arousal, while HR has been shown to correlate with valence [68]. With regards to signal variations in both HR and GSR (Figure 2). Thus, the learner’s affective state is determined by normalizing HR and GSR variations with regards to the baseline. For example, a very high positive HR signal and a very high positive GSR signal will be considered as excited. Normalization is done by mean-shifting (subtracting current values from the baseline and dividing the difference with the standard deviation). For readability purposes, we will refer to the mean-shifted normalized values simply as mean HR and mean GSR from now on.
5. Experiment
The current experiment is designed to build on previous results obtained with the use of a novel learning technique employing subliminal stimuli in a way which accelerated the deductive reasoning of learners during a learning session [19]. This experiment teaches the same lesson, learning how to construct a magic square in 3 simple tricks without the use of neither complex mathematical operations nor calculator, but within a very different environment and context. First, we are using a 2D system that looks very similar to an online exam session. The idea is to minimize learner’s distractions as much as possible compared to the previous 3D serious game environment cited above. Second, the solution to the three tricks was not presented here. The learners had to infer their own solutions and correctly figure out the algorithm used in each trick. Third, the context also differs for we ask learners this time to make the fewest amounts of mistakes possible whilst taking the shortest amount of time. In our original study, learners could roam around the 3D world freely and were not subject to any time constraints. The focus of this study remains the same: visually teach the construction of an odd magic square of any order with the use of neither a calculator nor one mental arithmetic operation. To construct a magic square, one must successively apply three simple tricks (see [19] for more details).
5.1. Methodology
The experiment consists of five steps depicted in Figure 3. The experiment went as follows: in Step , a series of neuropsychological tests were administered, namely, a spatial and pattern recognition memory tests (SRM and PRM, resp.).
(a)
(b)
Learners proceeded to Step where the three previously mentioned tricks were taught. We decided to show the learners multiple examples of each trick without explaining how the trick works. It was up to them to deduce the solution, thus the interworkings of each trick. Learners reported how they figured each trick by choosing between the following: I deduced the trick by intuition, logic, a little of both, mainly by intuition, or mainly by logic. It was clearly explained that answering “by intuition” should mean “after thinking about the available choices, a decision, judgment, or solution seemed correct in the absence of supporting evidence” [69].
When all the three tricks were deduced, learners were instructed, in Step , to respond to a series of 13 questions. No solution is given to the learners at any time while learning the tricks. The first series of 10 questions, Q1 to Q10 tested their knowledge of each learned trick. Questions 1 to 3 were directly related to the first trick, questions 4 to 7 to the second trick, and questions 8 to 10 to the last trick. The last 3 questions, however, tested their knowledge of all three tricks. Indeed, a partial magic square is presented and learners have to combine all three tricks to answer properly, by a yes or no, if the presented square is correct or not. A fixed time limit of 45 seconds for the questions was imposed. Failing to give an answer within the allowed time was considered a mistake. Learners reported how they answered each question by choosing between the following: I answered the question by guessing the answer, by intuition, by logical deduction, mainly by intuition or mainly by logical deduction. After giving their answer, a green check or a red cross appears for 2 seconds indicating to the learner if they made a correct or wrong choice, respectively.
After answering all the questions, a series of posttests were administered in Step to test for prime awareness and overall system evaluation, including difficulty levels and intrusiveness of the physiological equipment used (based on a Likert scale from 1 to 6 where 1 = not intrusive at all and 6 = very intrusive).
Lastly, Step displayed the current obtained results as well as information regarding time and performance.
Learners were organized into three distinct groups. A first group (control group) getting no subliminal cues at any time during the experiment. They did, however, get primes both masks just so EEG data would not be biased. Another group (answer group) will get primed with the answers for the tricks (Step in Figure 2) by the tutor (e.g., the primed cue for the multiple choice questions will be “flashed” over the square corresponding to the correct answer). The tutor will project subliminal misleading cues for the tricks and the questions for the last group (miscue group). The subliminal stimulus and thresholds were carefully chosen following the neural bases of subliminal priming [15]. Each stimulus was preceded by a 50 ms premask of random geometrical figures, a 33.33 ms prime (2 frames of a 60 Hz SVGA screen) and a 50 postmask of random geometrical figures. Prime and mask speeds were designed following methodologies previous reported [22, 46].
Each learner was compensated with 10 for their participation. We also added an extra monetary incentive by informing learners that an additional 20 will be handed out to the top 5 learners who committed the fewer amount of mistakes while taking the shortest time to complete the lesson. This incentive idea is inspired from recent studies showing that subliminal instrumental conditioning can take place in the human brain if proper conditions are met [17, 46]. We are hoping that subliminal cues (associated with positive answers) along with the monetary incentive will provide sufficiently strong conditions for the creation of an “unconscious association” between the stimuli and possibly the decision to choose the correct answer. The teaching material is separated into parts, or PowerPoint-like slides, and displayed at a slow rate to give every learner an equal chance at fully reading each “slide”.
A total of 43 healthy volunteers, 19 men and 24 women, took part of the experiment. The sample’s mean age was 27 (SD = 3.51). A repartition of the learners can be found in Table 1.
5.2. Cue Construction
In this experiment, cues are constructed to help guide the visual attention of the learner towards the constructions of the solution used. For example, trick number one requires one to place the next number one slot above and two slots to the right. One possible answer would be to prime the arrow seen in the left part of Figure 3. This priming is spatially aware, that is located at the place where we believe users will be looking and trying to figure out the answer. Thus, for the letter 10, the answer cue will be projected in a way that the base of the arrow starts from the letter 9 and not anywhere else. Miscue cues are created to distract learners and to possibly point them in the wrong direction. In this case, for example, the red arrow is pointing toward a completely false cell. The answer cues and miscues have therefore been tailored for each slide to specifically reproduce the described behaviour.
5.3. Physiological Recordings
Physiological recordings, namely, EEG, Heart rate (HR), and Galvanic skin response (GSR) and two webcams (mainly used for signal cleaning and noise reduction) were used throughout the experiment. An EEG base line of two minutes with eyes opened and eyes closed preceded all monitored activities. The EEG data was recorded from six active sites using an elastic cap at locations C3, C4, P3, and P4 as defined by the international 10–20 system [70] and referenced to Cz(Figure 5). The last two actives sites are A1 and A2 and are more typically known respectively as the left and right ear (referential linked ears montage). All recordings were achieved at a sampling rate of 256 Hz (4ms precision, reliability of reporting up to 64 Hz). To extract all the EEG band passes described earlier, all EEG signals were filtered using a band-pass Butterworth filter, therefore isolating Theta (4.5–7.5), Alpha (8–12 Hz), Beta (13–30 Hz) and the 40 Hz band. Matlab running under a 64 bit operating system was then used to compute and normalize the data. Most eye blinks and eye movement artifacts occur dominantly between 0,1 and 4 Hz band and thus were eliminated by a proper band-pass filter as well.
6. Results and Discussion
We want to investigate in this paper the possible impact of positive subliminal prime on learning and whether this impact can be observed in physiological recordings. Our findings will be presented in the following subsections.
6.1. Pretest Results
Before going into further details and explain our findings, we thought it necessary to check if significant differences were observed between groups of learners with regards to pretest and posttest assessments. Thus, no significant differences between groups of learners were found regarding the SRM and PRM pretest results (expressed in speed of response and percentage of correctness). These tests are important because they could have introduced an imbalance between learners, thus creating a bias for the upcoming results. Also, the results of the speed test both in terms of response time and accuracy were not significantly different between groups. Furthermore, assessing lesson difficulty and question difficulty are two important factors when determining a learner’s success rate. Once more, no significant difference was found between the three groups. Another post-test that could have has an impact on results and motivation is the lesson likability factor. Again, no significant results were found between users. Finally, we wanted to test whether the sensors were intrusive or not and disturbed the learners while learning. The Likert scales used for most of these test was from 1 to 6, 1 being “not at all” and 6 being “a lot”. The intrusiveness test for example asked whether the EEG headset hindered the learning process or disturbed then in any way (from 1 to 6). Results show that neither the EEG sensors nor affective sensors disturbance was in any way significant across groups. These initial results tend to show that the positive impact observed on learning seems is not biased by subjective factors such as reported likability or experienced difficulty.
6.2. Descriptive Analysis
The first research question we wanted to investigate was the overall performance of the subliminal group versus the other two (control and miscue). We ran a cross-tabulation analysis and found a significant interaction effect between group and overall mistakes (group mistake, , alpha = 0.05). Table 3 shows the distribution of the data. Each answer given is computed and presented here. Furthermore, the table is grouped by all type of responses. We did not distinguish between guessed answers or not.
Pushing the analysis further, we found a significant interaction with answer type, mistake, and trick answer: that is how did the learner actually comprehend the lesson versus how they responded to the question and the number of mistakes made (answer type mistake trick answer) with a , alpha = 0.05 when learners have understood the trick by “Logic”, for learning the trick by “Intuition” and when learners learned the tricks by using a mix of both logic and intuition. The results are presented in Table 4.
However, we wanted to investigate if cue type could indeed be responsible or have an impact on learning performance. We found a significant interaction effect between question answer type, mistake, and group (question answer type mistake group). This interaction, however, is only significant for the control group (, alpha = 0.05) and answer only (, alpha = 0.05). These results, shown in Table 5, seem to indicate that miscues could actually result in an almost dispersal effect of performance (sometimes high results and sometimes low but nothing statistically significant) whereas the positive cues seem to converge and positive increase performance for almost all type of question answer type when compared to the control group. One notable exception is the “mainly by intuition”, even if results are very close nevertheless.
These results seem to indicate that positive primes seem to enhance learning in several important areas, but pushing the analysis further and running a four way cross-tabulation analysis presented in Table 6, results show that there is indeed a significant effect of trick answer type, question answer type, group, and mistake (mistake question answer type group trick answer type) but only for the following combinations of trick answer and group:(i)logic control (, alpha = 0.05),(ii)logic answer (, alpha = 0.05),(iii)a little of both answer (, alpha = 0.05).
The results seem to indicate that positive subliminal cues do seem to strongly influence logical decision making when learning a trick logically. It was expected though to have the best results in this category. After all, the experiment was a logical learning paradigm with pattern recognition. It was not a drawing contest. We are, however, surprised to find that positive subliminal cues alone seem to cause almost a 10 increase in performance when learners use a little of intuition in deducing and answering questions.
Another important aspect to investigate in our study was the performance with regards to response time. We gave specific instructions to the learners to try to make the fewest mistakes in the shortest possible time. We ran an analysis of mean with variance (ANOVA) across subjects by taking into consideration the following variables in Table 7.
We also computed a success ratio based on three variables: number of mistakes, total questions response time, and total tricks completion time. The rationale behind this ratio is based directly on the experiment requirements. Indeed, the tricks were designed to be short with an average of 2 minutes per trick. The questions, however, were designed to be answered in a maximal time of 45 seconds each. The hardest questions, questions 11 to 13, should take no more than 30 to 40 seconds to answer correctly. Thus, a standard learner should be able to finish answering questions is roughly the same amount of time or less that the learner took learning. So, basically the division ratio should return a result around 1. In order to distinguish between learners having made the same amount of total mistakes, we simply added the total number of mistakes to that ratio yielding the success ratio. Furthermore, since we were looking for the lowest number of mistakes in the least amount of time, it seemed reasonable to have a ratio where the lower the value the better the success.
The analysis if means ran on the results show that there is indeed a significant interaction effect of success ratio and group (average of control group = 7.26, answer = 4.41, miscue 7.09 thus not much different than control). Table 8 presents all the other pertinent results.
It is not surprising though to not expect an effect of total tricks completing time with group because some learners will want to take more time to learn a trick than others. In fact, the time one takes learning, one trick is less important than the amount of mistakes one makes (in our experiment). The same argument is given to question response time. However, the lack of interaction between the total amount of mistakes and group for the first trick can be surprising at first. However, after further investigation, we can come to the conclusion that since this trick is really the easiest of all and that the subliminal cues can hardly misdirect learners and thus obvious that everybody should at least have a good score here.
In light of the descriptive analysis, we can claim that positive subliminal cues do seem to have an impact on performance if analyzed with a success ratio and per trick. Indeed, the further we advance in the lesson the harder the tricks become for they are cumulative.
6.3. Affective Data Analysis
The first research question we wanted to investigate was the impact, if any, of the subliminal cues on learner’s performance and problem solving capabilities. As previously mentioned, we present affective results, mainly IBI (inter-beat interval) and skin response that have been mean-shifted and compared between subjects. Figure 6 presents both results for the IBI as well as the skin response signals.
(a)
(b)
We ran a multiple comparison of means (MCM) on very point presented on each graph and the presented results are statistically significant, even if points seem to overlap. We ran those ANOVAs on five groups: answer, control, miscue, top5 (the learners with the 5 best success ratios), and bottom5 (the learners with the 5 worst success ratios). As an example, consider for example the IBI data for Trick 3 where the value of group Top5 (solid round blue line) is almost identical to that of Bottom5 (dashed triangle line). In this case, one can say that the values are close or almost identical but they are statistically significant nonetheless. The differences are sometimes so small that they appear as the same on the graphs.
The most important column to look at is the answer column. This is indeed the state in which learners a few seconds (5 seconds) before answering a question. We can see that conditioned learners seem to follow the same overall morphology of the curve representing the 5 best learners. A negative IBI and a positive skin response might suggest that positively conditioned learners seem to be the more relaxed of the groups when answering questions. Indeed, they seem to have quite a high skin response value and the highest IBI interval value which is not recommended for it seems to lead to an excess of attention or implication while resolving the question. Furthermore, the positive subliminal cues seem to help converge the answer group’s IBI values towards the top5 group values, which was not the case for the miscue group. However, this observed pattern does not seem to occur with skin response. Indeed, the subliminal primes do not seem to “push” the answer group towards the top5 values. These results deserve further analysis in light of the EEG data that will be presented next.
6.4. EEG Data Analysis
The EEG asymmetry results have been computed for the entire brain area by following the methodology described by Davidson [61]. These results have been computed by subtracting the left hemisphere from the right one RH − LH/(RH + LH). Thus, a value of −0.5 would represent a strong 50 left side asymmetry. It is important to mention that the same MCM was computed on every point displayed in Figure 7.
The EEG results tend to confirm the descriptive and affective data analysis when it comes to answering questions. Indeed, the asymmetry value for the 40 Hz answer group is significantly different than the miscue group. The surprising result comes from the asymetry values for the third and most difficult trick. The positively conditionned learnerd seem to shift their attention and “insightful” problem solving strategy closer to both hemispheres, thus involving a lot more the right side of the brain that have been known to be an important actor in insightful problem solving.
We separated the Beta band in two because neurological evidence presented by [52] seems to point towards the fact that a strong left hemispheric dominance of Beta2 can be an indicator of stress or anxiety while engaged in a learning activity. Indeed, we can observe that the bottom5 group is much closer to a left asymmetry than the answer group which seem to make sense with regards to their earliest results.
7. Conclusion and Future Work
We have discussed in this paper the use of subliminal priming for enhanced learning. We believe that we have shown that the new IUI implementing subliminal cuing can indeed be used in both a 2D and 3D learning environment and can lead to very encouraging and promising results. Indeed, the implications of this study for HCI developers and experimenters are diverse, but converge on five fronts. HCI system designers wishing to construct an environment for enhancing either the learner’s deductive or inductive capabilities are not bound by a 2D or 3D computer environment. They can integrate positive cognitive cues as part of any computer environment. The presented results from this study, combined with previous results [19, 28] seem to support this claim. The present study confirmed that only positive cues, not miscues, seem to show great potential in enhancing the learner’s inductive capability, notably intuitive correct guessing, in a logic-based problem solving task. This issue is important to consider, from an HCI standpoint, when constructing smart interfaces to interact with a learner. This result seems to complement results from previous work in the HCI community by DeVaul et al. [20] claiming that relevant overt cues (or miscues) may actually help improve performance. Results from this study show that irrelevant miscues do statistically neither hinder nor enhance the learner’s performance. Another surprising and important finding in this study is the seemingly cumulative effect of positive subliminal priming on the learner’s performance throughout the lesson. Indeed, our results seem to point towards the fact that the effect of positive cues seems to be proportionally related to the trick’s difficulty. This is especially the case when positively conditioned learners (primed with positive cues) learned a lesson with using both logic and intuition. An intelligent interface, throughout the EEG and affective physiological sensors, can easily detect such learning conditions with the use of specific EEG asymmetry variations in the left versus the right lobes and adapt their priming strategy accordingly. The use of positive cues seems to have an important impact on learning but more specifically on the learner’s affective state when answering the questions. We observe that conditioned learners seem to be in a more relaxed state, while staying aroused to stimuli throughout the lessons. The EEG asymmetry signals indicate that learners in that group seem to “shift” their problem solving strategy from purely logical to a more intuitive, even insightful, approach as indicated by the strong presence of 40 Hz waves. Such precious results and data are only obtainable through the use of EEG sensors and could become of great potential and interest for the HCI and ITS community. Indeed, Intelligent Systems that can detect these shifts in strategies can adapt the course, tutor strategies as well as difficulty levels to the current and exact mental state of the learner through a smart and much more interactive UI.
Although these results are more than encouraging, the experiment is still missing a few avenues of improvement. First, the need to implement in any learning environment, but more specifically one that deals with unconscious cognition, an input for user feedback regarding the “strength” or “confidence” of his answer. We have observed this attitude in the video feedbacks a few times. Learners getting primed with miscues consistently kept getting correct answers, or the inverse: learners getting primed with positive cues did not properly answer the question. Although the later occurred at a much lower frequency, it seems interesting to investigate the learner’s “trust” in their own answers. Second, the offline analysis is definitely something to improve on. An Intelligent system built behind a IUI should be able to make decisions based on a real-time analysis of physiological data processed by proper filters and signal processing algorithms. It is indeed one of our future avenues of research. And third, an in depth EEG analysis is required in order to examine other sources of information such as response time and ratios related to specific pattern recognition in order to help establish the optimal cerebral conditions for efficient subliminal learning to occur.
We are currently working on perfecting our new IUI by implementing signal processing algorithms enabling us to filter EEG and affective signals in real time. Furthermore, we wish to investigate other aspects of unconscious cognition, such as intuitive decision making, by implementing a multiagent architecture to model the learner’s subliminal and affective profile. This will hopefully enable our IUI and ITS, through the use of a fully trained neural network and subliminal cues, to determine the proper course of action to take in order to enhance learning performance, either deductive or inductive, with regards to the emotional, cerebral, and problem-solving ability of the learner.
Acknowledgments
The authores would like to thank the two anonymous reviews for their insightful and helpful comments. Furthermore, they would like to thank the Fonds de recherche sur la société et la culture (FQRSC) for funding this research.