Abstract
In engineering applications, human comfort fulfillment is challenging because it depends on several aspects that can be mathematically controlled and optimized, like in case of structural, energy, or thermal issues, and others. Major troubles can indeed derive from combined human reactions, which are related to a multitude of aspects. The so-called “emotional architecture” and its nervous feelings are part of the issue. The interaction of objective and subjective parameters can thus make the “optimal” building design complex. This paper presents a pilot experimental investigation developed remotely to quantify the reactions and nervous states of 10 volunteers exposed to structural glass environments. As known, intrinsic material features (transparency, brittleness, etc.) require specific engineering knowledge for safe mechanical design but can in any case evoke severe subjective feelings for customers, thus affecting their psychological comfort and hence behaviour and movements. This study takes advantage of static/dynamic Virtual Reality (VR) environments and facial expression analyses, with Artificial Intelligence tools that are used to measure both Action Units (AUs) of facial microexpressions and optical heart rate (HR) acquisitions of volunteers exposed to VR scenarios. As shown, within the limits of collected records, the postprocessing analysis of measured signals proves that a rather good correlation can be found for measured AUs, HR data trends, and emotions under various glazing stimuli. Such a remote experimental approach could be thus exploited to support the early design stage of structural glass members and assemblies in buildings.
1. Introduction
It is generally recognized that human comfort in the built environment is a target for a multitude of aspects [1,2]. Engineering tools that can be used to optimize comfort are able to account for a great number of factors and parameters which can affect comfort levels severely. Typical applications are focused on thermal comfort, indoor air quality, visual comfort, noise nuisance, ergonomics, vibrations, and others. In addition, it is also recognized that other aspects (like, for example, personal factors, nervous states, and architectural parameters) should be also properly considered in the list of influencing indicators for comfort. There are no doubts about the correlation of built environment characteristics and its impact on the occupants’ emotions, behaviours, and physical well-being [2]. The same behaviours can also affect typical engineering issues and indirectly influence the building design and detailing [3–6].
Structurally speaking, vibration serviceability issues and the consequent risk of user discomfort are, for example, usually addressed in terms of acceleration peaks and recommended limit parameters. Their satisfaction ensures possible annoyance for customers or suggests a design retrofit/modification [7–12]. As a matter of fact, however, there are no doubts that human reactions to vibrations strongly depend on the physiological perception of frequency and amplitude of vibrations. The operational context and the nervous state of users have thus a severe impact on the degree of human tolerance of vibration issues [9]. Most importantly, such a set of feelings can be maximized in presence of uncomfortable architectures and building components.
The well-known psychological effect of architecture can have both positive and negative effects on users and thus can evoke subjective feelings that could possibly interact with mechanical choices and technical detailing of structural design [13]. This is typical of architects that evoke nervous states in the so-called “emotional buildings” [14–19]. Pilot literature studies, in this regard, proved that experimental measures can be carried out to quantify users emotions [20]. Smart sensors for real or Virtual Reality (VR) walks proved also to represent useful tools in support of the quantitative analysis of human reactions and emotions to building stimuli [21,22]. In this paper, a special care is spent for structural glass applications in buildings. Known as versatile but vulnerable constructional material, glass transparency and capacity to adapt to various setup configurations make it a largely used solution [23]. The high aesthetic impact of glass structures can be thus sometimes in contrast with the need of more efficient feeling of protection for the occupants, as it could be for extreme accidents, pedestrian systems, or uncomfortable configurations (Figure 1). Glass walkways are in fact often known as “architectures of vertigo” [26], where transparent structures are conceived as spaces of visceral thrills and intense psychophysiological stimuli with deep sensory experience and sociospatial implications. Additionally, glass pedestrian systems can be highly demanding in engineering terms, due to the mutual interaction of mechanical dynamic parameters, complex Human-Structure Interaction (HSI) phenomena, and even severe subjective reactions [9]. The analytical analysis and quantitative measure of emotional states for customers can thus represent a useful tool in the whole design process.
(a)
(b)
(c)
2. Research Goals and Experimental Strategy
A virtual experimental study is presented and assessed in this paper, in support of structural glass design. The goal of the proposed method is to quantify the prevailing human reactions and comfort levels for occupants that are exposed to different virtual built environments made of/with a primary role of glass. The measured reactions are then compared to address the psychological comfort level of volunteers under imposed visual stimuli. In accordance with Figure 2, the process takes advantage of a quantitative analysis of emotions based on the analysis of facial microexpressions (Action Units (AUs) of nervous states) and also heart rate (HR) parameters for volunteers exposed to several glass built scenarios.
(a)
(b)
While the use of glass material in buildings is largely increasing and further demanded in the post-Covid-19 stage [27,28], its impact on the psychological comfort of customers can be severely affected by transparency, intrinsic brittleness, lack of technical knowledge on its structural performance, vulnerability risk, etc. [29,30].
Glass material for structural applications is in fact known to require specific engineering knowledge for optimal and safe mechanical design. Besides, discomfort for building in occupants can still arise and thus affect their behaviour and movements inside/onto a glass system. Typical examples can be related to glass pedestrian systems or structural glass barriers with risk of shards/fall.
In this regard, under the limits of remote experimental measures, and based on the herein presented research outcomes, it is expected that building structural design could take advantage of subjective measurements in support of classical mathematical tools and engineering models. Besides, real in situ experimental procedures should be also carried out for a further extension of the methodology.
More in detail, the pilot experimental investigation was carried out with the active contribution of 10 participants and the FaceReaderTM automatic facial expression recognition software [31]. The number of volunteers was chosen based on the availability of individual participants and on the number of experimental records to process for each participant/stimulus/time interval. The group of participants was selected from a class of volunteers preliminary involved in measurement trials discussed in [25]. The class of volunteers included 60% females and 40% males, with an average age of 28 years. No preliminary technical knowledge on structural glass in constructions was required at the time of the experiment.
The remote experimental procedure was planned as in Figure 3, as an adaptation and extension of [25]. Two different stimuli were designed to address nervous states and human reactions of volunteers, namely, a VR set of “static” items and a “dynamic” video clip of virtual walks. A special attention was paid for the selection of stimuli, so as to capture different facial microexpressions and emotions. For this reason, a preliminary Computer Assisted Web Interviewing (CAWI) survey was also distributed in Winter 2020 and used to explore the prevailing subjective reactions for the selected participants asked to virtually deal with buildings or open spaces characterized by a prevailing role of structural glass. At the end of the remote experiments, moreover, the participants were asked to comment about their feelings.
There are several literature studies which confirm the potential of human emotions as a guide for designers in different research fields [32,33]. Besides, such a result can be achieved in different ways. Facial microexpressions are, for example, considered essential for the quantitative analysis of basic emotions, because human faces provide useful information about feelings and the inner individual states. At the same time, there is a large number of literature experiments that confirm the strict correlation between human reactions and emotions with HR data and their variations over time [34–36], as well as the useful feedback that can be derived from biosignals [37–39]. In the present study, both facial expressions and optical HR data are recorded for volunteers exposed to virtual glass structures.
As shown in Sections 3–5, the analysis of experimental results confirms that the use of glass in buildings is still largely affected by scattered human reactions. For selected stimuli, a rather close match can be observed for most of participants. It is worth to be noted that a good trend in facial microexpressions/emotions and HR data can be found for the imposed scenarios. Such an outcome, consequently, suggests the comparative analysis of remote facial measures with possible in situ experimental records.
3. Virtual Experimental Analysis
3.1. Remote Testing
The procedure schematized in Figure 4 was repeated twice for each volunteer, first with a static VR input and later with a VR video clip of virtual walks. All the experiments were carried out remotely, under Covid-19 restrictions to mobility, with the support of laptops, a shared screen for the input stimuli, and a web cam for recording the face of participants.
The invited subjects were preliminary informed about how the experiment would be conducted. To this aim, the setup was properly checked, so as to avoid disturbing effects for the record acquisition and for the postprocessing analysis. A special attention was given to ensure an optimal and mostly uniform ambient illumination and face exposure for the participants. The position of the screen for each participant was also checked, so as to avoid distortions or introduce misleading inclination values for the experimental acquisition. Finally, the presence of participants with glasses or beard was allowed but separately noted, for a more refined analysis of collected data. Once the operational conditions were verified, the input source was shared to all the participants. During the presentations, audio communication was allowed to provide additional comfort to the participants. No audio source was indeed introduced in addition to the visual input. During each one of the presentations, the webcam of controlling machine was used to record the facial expressions of the participants, while looking at the shared screen. For this purpose, a preliminary tentative to capture the best cam resolution and sampling rate was carried out. The final choice resulted in high quality records with minimum 1280 × 720 (or 1920 × 1028) video resolution and a total of frames in the range of 4300 fo each signal (30–60, the range of frame rate; 12432 kbits/s, the average bitrate). The software analysis was based on all the collected frames for each record, thus requiring an average of 10 minutes/setup.
3.2. Measurement of Raw Signals
The analysis and interpretation of facial expressions and heart rate data for the participants exposed to VR stimuli was carried out with a hybrid approach. A fundamental role was assigned to the automatic analysis offered by FaceReaderTM, for the quantification of basic human reactions, emotional states, and HR data as in Figure 2. Furthermore, the manual elaboration of software results represented a key step of analysis.
The selected commercial software was used because it is based on Artificial Intelligence tools and can efficiently detect subjective emotions, human reactions, and HR parameters from input video records. Based on 500 key points for the description of facial movements, the output classifications of N = 7 nervous states based on facial expressions include “happy”, “sad”, “scared”, “disgusted”, “surprised”, “angry”, and “neutral” feelings. These states are normalized and quantified in terms of instantaneous evaluation of Action Units (AU) for basic emotions. Among others, an additional intrinsic advantage of the used software is represented by the optical measure of HR of subjects, based on complex remote photoplethysmography (rPPG) optical techniques. The rPPG module analyzes HR data and HR variability from input videos of participants, by quantifying the amount of light that is reflected by their face. The latter relates to cardiac cycles and changes in blood volume based on video captured by the camera [40]. Different literature studies proved that the remote HR analysis can be powerful and efficient [41–44]. The limit requirement is that setup conditions for the experiments are properly addressed. Possible movements of participants (i.e., while walking, talking, etc.) may result in overestimates or underestimates for real HR data [45,46].
In the present study, the implicit advantage was represented by constant ambient conditions for all the participants and by their mostly fixed position during the whole experimental analysis (sitting on a chair). Moreover, a preliminary check of setup features resulted in input video records for the software generally classified as “high quality” video sources for remote analysis (Figure 5). An example of resulting AUs and HR records is shown in Figure 6 for one of the involved participants. The experimental measurements for each participant/virtual experiment/inner state were in fact collected in the form of AU charts of single basic emotions (in the range from 0 to 1) and HR data (bpm), as a function of time.
3.3. Postprocessing Strategy for Measured Raw Signals
A set of charts agreeing with Figure 6 was used for the interpretation of human reactions and for the analysis of their possible correlations with the imposed VR stimuli. To this aim, postprocessing feedback from volunteers was also used to double-check the emotional states. It is worth to be noted that the chart in Figure 6 omits the first instants of records, where a preliminary discussion was carried out with each participant to provide instructions and create a comfortable condition.
3.3.1. Action Units and Emotions
Given that multiple AU records were collected for each participant/state/VR stimulus, the first postprocessing step consisted in the detailed analysis of single AUs. The first effort was spent for the normalization of AU records in time, so as to offer a more independent analysis and comparison of AU records for multiple participants. This was measured aswith 0 ≤ t ≤ 120 s and AUnorm(t), the normalized record for each participant, emotion, and VR scenario, at the time instant t; AUoriginal(t), the original signal record for each participant, emotion, and VR scenario, at the time instant t; AUinitial,avg the mean value of each emotion, participant, and VR scenario (preliminary stage).
Following equation (1), the individual records of basic emotions, AUnorm(t), were successively grouped in basic pos(t) and neg(t) plots. In the pos(t) set, the AU data in time for feelings marked as “happy” were considered only:
Similarly, the neg(t) set included the input data from feelings marked as “sad”, “angry”, “scared”, and “disgusted”:
The “surprised” AU data (“SUR(t)”) was disregarded in the so-calculated pos(t) and neg(t) signals but analyzed separately. Depending on the individual subject, context, and stimulus, “surprised” states can be in fact typically associated with both positive or negative feelings [25]. For the same reason, “neutral” states were omitted from the study, due to the virtual remote setup. The final marking of emotional data was based on the analysis of prevailing AUs at a given time instant t, where the SUR(t) input was superimposed asor
In conclusion, the final detection of comfort trends for a given visual stimulus was carried out based on the percentage variation of elaborated signals, over the time of analysis. At each increment t of the experiment, the emotional variation was calculated for both POS(t) and NEG(t) data aswith Y = POS(t) or NEG(t), respectively, as in the example of Figure 7.
(a)
(b)
The prevailing emotional state (and thus comfort trend) was established analytically, in terms of absolute maximum values of nervous feelings and max/min percentage variations. Each time instant t was classified with a final POS or NEG mark, uniquely detected from data elaboration. A unitary “comfort weight” value C(t) = POS or NEG (based on the prevailing nervous state) was assigned to each participant/instant/scenario, to facilitate the comparative analysis of multiple signals and stimuli. More precisely, for static VR, the primary/prevailing emotion based on AUs was calculated for each one of the 27 input items and for dynamic VR input, the attention was still focused on primary emotions based on AUs, but for specific frames only that were detected and selected from absolute AU peaks and building configurations.
3.3.2. Heart Rate and Emotions
The elaboration of HR data derived from rPPG techniques was based on the analysis of single measures over the time of the experiment. Studies of literature (see, for example, [21] and Figure 8) demonstrated that HR measures can offer useful feedback for the quantification of emotions and human feelings. In the present study, based also on the relatively short duration of stimuli, the attention was focused on the comparison of HR data trends with facial expression measurements, for a given participant/stimulus. In the present study, the HR variation in time was labelled as HR+(t) or HR-(t), respectively (with 0 ≤ t ≤ 120s), withor
(a)
(b)
As in Section 3.3.1, a unitary “comfort weight” C(t) was assigned to the HR+ and HR− parameters, so that each instant t could be quantified as in equation (7) with a POS or NEG reaction. Care was paid especially for HR+ data, so as to detect possible discomfort/NEG feelings of volunteers, while the HR decrease (HR−) was related to comfortable situations. Furthermore, a null comfort weight was assigned to the limit condition HR(t) = HR(t-1).
3.3.3. Correlation of Heart Rate and Facial Expression Measurements
The combined analysis of AUs and HR outcomes for the comfort quantification of participants was carried out by comparison of normalized weights over the time of the experiment. In this manner, the attention was focused On the analysis of single elaborated signals for a given participant and stimulus (HR or facial AU) On the synchronized analysis of HR and facial expression data for a single participant, over the time of experiment; in this case, the attention was focused on the potential agreement of AU and HR data and trends at a given time instant t On the analysis of reaction trends from the whole group of volunteers
4. Discussion of Stimuli and Results
4.1. Static VR Input
The participants were subjected to a selection of 27 pictures, equally spaced at time intervals of 5 seconds. Figure 9 shows some examples, where labels #n denote the order of items. For each subject, the stimuli sequence was kept fixed. The items were selected from magazines, scientific journals, construction companies webpages, and newspapers. Major advantage for the selection of items was taken from the preliminary CAWI survey (Step 1 in Figure 3).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
4.2. Analysis of Prevailing Reactions Based on AUs and HR Data
Average POS and NEG reactions were first calculated for each static item and participant. Figure 10 shows the prevailing reactions in the group of volunteers, giving emphasis on measured (a) discomfort (NEG and HR+ data) or (b) comfort (POS and HR−), respectively.
(a)
(b)
As far as a different time instant is selected in Figure 10, it is worth of interest to note the variation of POS/NEG and HR+/HR− data for volunteers. The chart shows a clear variation of emotions and human reactions for an assigned stimulus, with marked fluctuations of POS and NEG data. Rather balanced POS/NEG conditions can be noticed for some instants of the experiment. Moreover (at least for some intervals), a good correlation of NEG/HR+ trends can be observed, in the same way of POS/HR− variations in time. Within the limitations of the study, such a finding confirms that HR increases corresponding to discomfort and negative feelings for participants, while the HR decrease can be related to a more comfortable situation and nervous state.
A more concise analysis of measured AUs and HR data for each stimulus can be indeed carried out based on Figure 11, where average experimental outcomes are proposed for the 27 items (5 seconds each). The good match of NEG and HR+ data for most of the pictures is worth to be noted. Rather close agreement (but for a limited number of items) can be observed also for POS and HR− trends, thus enforcing the potential of the procedure.
In this regard, some first conclusions about the experimental method can be derived when global results are grouped and analyzed as for items characterized by prevailing NEG reactions (#14, #19, #25, #27), prevailing POS reactions (#1, #10, #20, #21), mostly “balanced” NEG and POS reactions (#2, #3, #5, #15, #16), prevailing NEG values with a good match of HR + reactions, to denote discomfort (#19, #25), and prevailing POS values with a good match of HR + reactions (#10, #20, #21, #24).
Selected items with marked NEG and POS reactions are proposed in Figures 12 and 13, respectively.
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
4.3. Analysis of Reactions Based on the Subject of Items
A special care was paid for the analysis of experimental measurements for grouped static pictures. The attention was focused on glass structures and components representative of major categories of items and built environments, namely: Group A: pedestrian glass systems and structures or glass elements characterized by risk of fall for the occupants (items #4, #8, #10, #13, #15, #17, #18, #20, #22, #23, and #27). Group B: glass systems or elements with damage and/or under hazard (items #6, #9, #12, and #25). Group C: others (#1, #2, #3, #5, #7, #11, #14, #16, #19, #21, #24, and #26).
Figure 14 reports the residual static items from the full set of 27 pictures, while Table 1 presents a summary of prevailing reactions based on facial expressions and heart rate parameters (% of prevailing measurements for grouped items).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
It is worth to be noted that there is a marked NEG response for groups A and B that are characterized by a major risk for the participants. In particular, the presence of damage (B) evokes high discomfort in the majority of the involved volunteers. Besides, for the set of pictures in group C, a more balanced response was measured. For the pictures in group A, the experimental outcome suggests that pedestrian glass systems involve the highest human reaction and intrinsic discomfort, compared to other glazing solutions in buildings. The effect can be justified by the fact that the human interaction with the structure becomes predominant. It is important to notice the POS effect of occupants in the pictures, as, for example, in the case of item #10.
Even under the limitations of the experimental study herein presented, the collected results suggest that as far as the structural design of the glass system is optimally designed to preserve robustness, stability, and overall protective feedback for the occupants, the first human reaction is slightly on the side of POS feelings. In this regard, it is necessarily required to explore further this kind of scenario and extend the VR experimental stage with field experiments, so as to capture the human reactions of active pedestrians and combine these emotional feelings with the structure vibrations and other mechanical parameters of primary interest for the dynamic characterization of the structure itself under the effects of walking occupants.
In terms of HR+ and HR− correlations in Table 1, finally, the collected results suggest some agreement with facial expression outcomes but also recommend the need of a more extended analysis in this direction (i.e., number of volunteers/stimuli).
5. Discussion of Results from Dynamic VR Experiment
The second stage of remote investigation was carried out with the same group of participants exposed to a virtual walk in a glazing environment. Major benefit was taken from the adaptation of a VR clip representative of a real case-study building (Generali Real Estate French Branch) located in Paris. The building, as shown in Figure 15, is currently under renovation and is characterized by a large amount of structural glass in facades, roofs, and floors. In order to preserve a total duration of 120 seconds for the whole experimental analysis, the selection of clips for the case-study building was divided into three different virtual walks (40 seconds/each), realized at the roof/terrace level (W1), at the ground level, from the main entrance of the building (W2), and at the first level (W3), within the internal open space. More in detail, the walks were characterized by the following: W1 (outdoor): risk of fall (glass balustrades) and presence of glass facades. W2 (indoor): risk of fall (glass floor), but also glass roof on the top, glass walls, and internal partitions. W3 (indoor): risk of falls (glass balustrades) and presence of a glass floor (visual stimulus only).
(a)
(b)
(c)
(d)
(e)
(f)
Figure 16 shows the analysis of POS, NEG, and HR+ data for the group of volunteers. The dynamic VR scenario gave evidence of scattered emotional responses from the volunteers. Furthermore, the HR+ trend for the group of volunteers was observed to have close correlation with NEG data, thus confirming and enforcing, under the limitations of the present setup, the experimental findings from Section 4.
Worth of interest in Figure 16 is the local and global fluctuation of POS or NEG peaks (and thus HR+ data) and their correlation with visual stimuli. Major NEG/HR+ peaks of discomfort were in fact detected for conditions characterized by the presence of outdoor or indoor balustrades (risk of fall) and floors (risk of fall),
while major comfortable responses were calculated for scenarios with facades/walls (no direct contact) and roof components (no direct contact),
thus suggesting again an agreement with the experimental outcomes from the static VR scenario.
On the other side, a more scattered and weak evolution of HR+ data with NEG values was noticed in Figure 16, compared to Section 4. A reason for such an outcome could be found in the dynamic nature of the stimulus itself [20,21].
Certainly, a direct comparison of human reactions under static or dynamic VR stimuli is not possible. The context features as well as the subjective response of participants in a given remote scenario could be affected by several aspects. In Figure 17, to this aim, the analysis of heart rate parameters is shown for all the volunteers exposed to the two experimental conditions. A mostly uniform response under static or dynamic stimuli can be observed only for a few participants.
(a)
(b)
In this regard, future investigations will be carried out to quantify more in detail the subjective reaction of occupants in glass building scenarios, as well as the correlation of design parameters and quantitative data of human reactions. Most importantly, instrumented experimental configurations will be examined under field scenarios with real walks from the participants and a new calibration of their subjective feelings.
6. Conclusions
The analysis, quantification, and optimization of human comfort in the built environment is a target for several design fields and depends on several engineering aspects that can be mathematically controlled to satisfy specific performance indicators. Typical examples relate to structural, energy, and thermal design issues. On the other side, human reactions are also sensitive to a multitude of parameters that find origin in subjective feelings of users, in the same way in which the so-called “emotional architecture” aims at evoking positive responses of building occupants.
In this paper, an experimental analysis was presented and addressed in support of the quantitative measure and analysis of emotional and nervous states of users for selected built scenarios, towards the definition of enhanced design strategies that could combine engineering targets/methods with architectural concepts, technological solutions, and evocation of subjective feelings of customers. In doing so, major advantage was taken from the use of virtual experimental techniques and a facial expression commercial software able to capture both microexpressions and nervous states of participants (based on facial Action Units (AUs)), as well as heart rate (HR) parameters. A group of 10 volunteers was in fact exposed to visual stimuli of static and dynamic glazing environments.
By designing an input stimulus characterized by a prevailing role of structural glass elements (i.e., roofs, floors, balustrades, etc.), the attention was focused on the quantification of the herein called “positive” (POS) or “negative” (NEG) feelings of participants based on their facial microexpressions and feedback, as well as to address the possible correlation of heart rate modifications with the so-detected POS and NEG emotions. As shown, the analysis of remote experimental measurements gave evidence of some good correlations of facial expression parameters, HR trends, and input stimuli, and this was found in agreement with preliminary expectations at the time of the experimental planning. Postprocessing feedback also suggested the use of remote tools and facial measurements in support of comfort analysis, as a direct quantitative measure of nervous states evoked by virtual stimuli. The remote measurement of HR variations for participants was then addressed to explore their potential correlation with selected input stimuli and corresponding facial expression/emotional state. Well promising outcomes were observed through the postprocessing analysis, with general HR trends in correlation with facial microexpressions outcomes, for most of the stimuli. On the other side, while proving the efficiency of the approach, the same experimental outcomes highlighted scattered human reactions for some of the stimuli, thus enforcing the need of additional in situ experimental procedures to address more in detail the emotional state and trend of glazing customers. [49].
Data Availability
The research data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare no conflicts of interest in the publication of this research study.
Acknowledgments
Seretti Vetroarchitetture S.r.l. (http://www.seretti.it) is acknowledged for sharing the dynamic VR video clip for the case-study building in Paris (courtesy of © Generali Real Estate French Branch, adapted from http://www.helloworldparis.com). All the participants that voluntarily and actively contributed to the experimental investigation are warmly acknowledged. Special thanks are due to Ms. Corine Tetteroo (Noldus Information Technology bv, NL).