Abstract
The Harmonic Walk is an interactive physical environment designed for learning and practicing the accompaniment of a tonal melody. Employing a highly innovative multimedia system, the application offers to the user the possibility of getting in touch with some fundamental tonal music features in a very simple and readily available way. Notwithstanding tonal music is very common in our lives, unskilled people as well as music students and even professionals are scarcely conscious of what these features actually are. The Harmonic Walk, through the body movement in space, can provide all these users a live experience of tonal melody structure, chords progressions, melody accompaniment, and improvisation. Enactive knowledge and embodied cognition allow the user to build an inner map of these musical features, which can be acted by moving on the active surface with a simple step. Thorough assessment tests with musicians and nonmusicians high school students could prove the high communicative power and efficiency of the Harmonic Walk application both in improving musical knowledge and in accomplishing complex musical tasks.
1. Introduction
Listeners perceive the surface of a tonal composition as a sequence of discrete pitch-events [1] and can extract the underlying harmonic background only thanks to complex cognitive processes which involve discontinuity and similarity perception, musical entities categorization, and events temporal organization. It is probably the reason why harmony has always been considered a difficult matter to understand, also among professionals. Nevertheless, harmony occupies a very steady place in human perception. A lot of research in the field of music psychology has offered evidence that unlearned adults and children from the very early age of 4-5 years [2] have wide implicit harmonic knowledge comprehending elementary chord functions detection and the perception of regularities in chord occurrences. As tonal music is ubiquitous in our music culture’s environment, all these features are learned from mere passive exposition [3] and can so be considered as a cognitive heritage common to every unskilled user. Thus, the main goal of the Harmonic Walk1 is to offer to people the possibility of experiencing such important musical knowledge in an easy and handy way, contributing to make users aware of tonal musical features (see Figure 1). The application, taking advantage of the large potentialities of a body moving in space, can provide the opportunity to understand the structure of a tonal melody, to explore the harmonic space, and to experiment chordal progressions. Moreover, it can return to users a physical feedback of the spatial relationships which govern the music chords domain, thus helping them to build their personal cognitive map of tonal harmony. The Harmonic Walk, matched with floor projections, can be used for edutainment purposes in demonstration setups as well as in sound and image installations. In schools it can be seen as a didactic tool to support music education programs.
(a)
(b)
(c)
The aim of this paper is to illustrate the Harmonic Walk system and to provide an initial assessment which can outline some interesting points of discussion about the application and its utilization in a didactic context. The paper is organized as follows: Section 2 reports and discusses some similar applications. Section 3 introduces the two theoretical principles upon which the application is based, namely, the fundamental process of tonal melody segmentation (Section 3.1), and the six roots’ harmonic space by which it is possible to harmonize a one-key tonal melody (Section 3.2). Section 4 addresses the core question of the Harmonic Walk application: the cross-domain mapping between the theoretical representations of musical features and their rendering in the physical space through a geometric interpretation. Section 5 presents the system architecture, describing the Zone Tracker application and masks for space subdivision, the Max/MSP modules for sound production, and the current implementation of application’s interactive interface. Then Section 6 follows where the three phases of the assessment tests are thoroughly described and in Section 6.6 quantitative and qualitative results are discussed. Conclusions and further work are outlined in Section 7.
2. Related Work
The idea of interacting with harmonies through spatial representations is not new. It has been widely developed in the Harmony Space project at the Music Computing Lab of the University of Stanford in 1994 [4]. The Harmony Space interface shows a desktop bidimensional matrix of pitches ordered by major thirds on the horizontal axis and by minor thirds on the vertical axis. Choosing a key area, a chord size, and a chord mapping, when a note is selected, the chord built on it will sound. The interface has been used to simplify the study of harmony, to analyze musical pieces, and to compose new ones. The environment is very rich and complex as it allows many option possibilities (the number of pitches used for the chord, the chord type, the key, etc.): so the interaction actually depends on a preselected series of options, which rather fits an expert user level. More recent systems like Isochords [5] or Mapping Tonal Harmony [6] are aimed at helping the understanding of the harmonic structure through a visualization of the space of musical chords or highlighting preselected harmonization possibilities. Again, these are very complex environments which improve musical structure consciousness at a high degree of knowledge.
All the interfaces till now described employ some visuospatial schemata of the harmonic space, but these are always reproduced on the computer traditional bidimensional screen. Nevertheless, Holland himself in 2009 tried a physical space extension of his Harmony Space employing a floor projection and a camera tracking system [7]. A GPS system to navigate a wide tonnetz2 area has also been proposed by Behringer and Elliott [8], where the authors suggest some musical games about composition and harmony features knowledge. The PaperTonnetz [9] presents a more intuitive approach, where chord progressions are obtained through a pencil navigation on an interactive paper plotted tonnetz. A more recent experience of interaction with Holland’s Harmony Space has also been experimented using a 3D graphical representation of the tonnetz. The user interacts with the interface through colored controllers which are used to select the pitches and to produce the chordal audio feedback [10]. Also in the Harmony Navigator [11] the chord selection, supported by a corpus-based statistical model, is operated by hand gestures in the 3D space around the user. All these experiences show that the role of the physical space in the tonnetz navigation is not irrelevant. “Deeper engagement and directness, rich physical cues for memory and reflection, embodied engagement with rhythmic time constraints, hands which are free for other simultaneous activities (such as playing a traditional instrument) and qualitatively new possibilities for collaborative use” are the benefits of physical interaction outlined by [7]. Others (Bigo et al. [9] and Manaris et al. [11]) emphasize the role of gestures as the preferred interaction style in the chords navigation, because they can help with the engagement of nonacademic musicians or amateurs with the world of harmony (Hedges and McPherson [10]). Lastly, the link between physical motion and the exploration of proximities in chord areas is the main point of Behringer and Elliott [8]. This is also the relevant point of the Harmonic Walk application, which has an immediate predecessor in the Stanza Logo-motoria [12], where the child has to use her/his spatial memory to match sounds with a spoken text. Some preparatory studies about orientation and wayfinding in the Harmonic Walk’s environment have been presented in [13], where assessment tests, carried out with a sample of children from 7 to 11 years old, showed that the majority of them was able to find a valid path to perform a harmonic progression. The Harmonic Walk presents a very simple harmonic palette restricted to only six roots without any further option. This aspect is much simpler than Holland’s Harmony Space. The roots spatial disposition on the actual surface, derived from a geometrical interpretation of their relationships, allows the user to interact with the roots simply moving on the surface, bypassing so any technical approach. By involving the whole body in the musical interaction the user has to activate harmonic perception, spatial memory, enactive route finding, and rhythmic skills in a measure much greater than interacting with a bidimensional computer screen. Moreover, this very basic setup projects its computational content in the real world, helping to bring the user nearer to complex musical concepts, like tonal harmony.
3. Theoretical Background
The Harmonic Walk is a theory-based application concerning the tonal music domain. Hence, the definition of the tonal music’s main features is fundamental to shape decisions about the system’s architecture and user interaction design. In Section 3.1 the processing of a tonal melody is analyzed in order to outline its schematic representation and to focus on its main components: the melodic segmentation and its underlying harmonic frame. Furthermore, in Section 3.2 the harmonic space of a one-key tonal melody is presented.
3.1. The Tonal Melody Representation
As soon as a listener is presented to a tonal melody, s/he first tries to interpret the sequence of notes, grouping them after a metrical and harmonic frame [14]. This produces a segmentation of the composition into different harmonic regions which, in case of one-key melody, shall all belong to the same tonality.
The example in Figure 2 shows an excerpt of a popular melody written in the tonality of F major. The gray units correspond to the tonic regions (T), while the violet ones correspond to the dominant regions (D). In the lower part of the figure, a one-line staff shows the durations of the units, which represent the harmonic rhythm of the composition. As can be seen, it is not regular even in such a simple piece of music. The perceptual, cognitive procedures which lie under this musical representation have been studied and formalized by Lerdahl and Jackendoff [15], who also showed how a tonal piece perception is organized at various levels, starting from the most superficial (grouping structure) to arrive to the deepest (prolongation reduction). Some of these level are shown in the upper part of Figure 2, where, from the 1st level to the 2nd level, a time span reduction is operated, based on the fact that the first dominant unit arrives on the weak part of the first bar. So, it can be incorporated at the upper level, creating a larger tonic unit which occupies two bars. This fosters the idea of the great importance of metrical accents, which rule the changes of harmony in a recursive way. Summarizing, a tonal piece may be seen as a sequence of musical units, segmented on the basis of their perceived underlying harmonic functions. The sequence has a metrical timing organization, which means that the region’s length is always a multiple of a basic time unit. Moreover, the chord sequence is also biased by the importance of primary chords and by the supportive role of their parallels. It has a strong recursive nature which leads mostly to prefer the tonic and dominant chords.
3.2. The Harmonic Walk’s Chord Space
The tonal language is a highly hierarchical system based on the prevalence of the tonic chord matched by the dominant (V degree, one fifth above the tonic) and the subdominant (IV degree, one fifth below the tonic). These three chords are said to be primary because they represent the fundamental tonal functions in a given key [16]. Although every one-key melody can be accompanied using only primary chords, also parallel chords, built one third below primary chords, can be used as supportive harmonies. Hence, the complete table of musical chords used by the Harmonic Walk application consists of six chords, corresponding to the first six degrees of the key3, as showed in Figure 4.
The tonal harmonic space has been historically represented by the tonnetz. In this representation, pitch is disposed in horizontal and diagonal lines. As shown in Figure 3, the horizontal lines connect the circle of fifths. On the diagonal axis SE-NW lays the major third circle while the diagonal axis SW-NE bears the minor third circle. The chord is represented by the triangular area resulting from the crossing of the three axes. The tonnetz representation shows the strict connections among the chords belonging to the same key. If we consider the whole chord progression comprehending the six C major chords from the left to the right (D minor-F major-A minor-C major-E minor-G major), we can see that they all are originated by the so-called edge transformation. The edge transformation is an inversion of the triangle along one of its edges, which means that two chords sharing the same edge have two pitches in common. Particularly, this relationship links each primary chord with its parallel (same color in Figure 3), while primary chords (F major-C major-G major) are originated by a vertex transformation, which expresses the commonality of only one pitch. All these observations can explain why the tonnetz representation of the harmonic relationships is so popular and meaningful also from the perceptual point of view. We shall discuss the six primary and parallel chords’ spatial disposition in Section 4.2.
4. From Theoretical Representations to the Physical Space: A Cross-Domain Mapping
Music is often said to be the “art of time,” as we have just seen that a tonal composition unfolds musical units in sequence along a timeline. Moreover, the concept of time involves musical memory, through which we perceive pattern recursion and formal organization. Nonetheless, the internal structure of musical elements has also a very strong spatial dimension. When we discuss one of the most basic musical structures, the scale, we commonly involve such terms like “ascending” and “descending,” which are obvious spatial metaphors. In expert speech as well as in common people’s language, melodies “go up or down” or “jump” or “arrive” to a particular point. Tones can proceed by “skip” or by “step,” keys can be “near” or “far,” harmonies make “turnarounds,” and bass lines are sometimes defined as “walking.” Music metaphorical schemas relating to embodied cognition and movement have been widely analyzed by Brower [17], Saslaw [18], and Cook [19]. Indeed, even in musical notation a single tone is represented as a point (a square “punctus” in Gregorian music or a circle, a “note” in the modern notation). However, we all know that every tone is not composed of a single element, but, rather, it is the result of the summation of a weighted number of overtones. So, every harmonic sound’s inner nature is a manifold whose single elements are organized after a precise spatial relationship: that of the harmonic overtones, to which the tonnetz, discussed in Section 3.2, owes a lot of its theoretical foundations4.
Now, to represent all this apparatus into a physical space, we need to provide a geometric interpretation of the tonal composition’s main features, which are(1)the musical regions sequence,(2)the six root progressions.
4.1. A Geometric Interpretation of the Musical Regions Sequence
The time proceeding of the various musical units is led by the melody, whose metaphoric scheme is expressed by the so-called “source-path-goal” schema [20]5. So, the description of a melody could include a starting point (the source), a series of intermediate steps (all the subsequent musical units), and the end of the musical phrase, usually a cadence to the tonic or to the dominant (goal). Following this metaphor and imagining the simplest motion in space a human can do, the walk, we could represent the tonal composition as in Figure 5, where each step corresponds to the next musical unit.
Hence, the geometrical interpretation of the “source-path-goal” schema is a sequence of units disposed along a straight line after which the user can perform the unit sequence till the cadence. To rebuild the piece’s correct time sequence, her/his pace has to follow the harmonic rhythm of the composition.
4.2. A Geometric Interpretation of the Six Root Progressions
As far as concerns the second musical feature, we need a geometrical interpretation of the harmonic space already defined by the tonnetz along the red, blue and green axes of Figure 3. If we look at Table 1, we see that all the chords, but the problematic vii degree, can be linked with more or less probability with all the others.
This means that, in spite of all its expressive power, the tonnetz is not the best spatial schema to be transferred to the application’s actual surface, because it does not allow moving from one chord to the other without touching some other chord. Instead, a user must be able to connect all the chords to realize all the possible harmonic progressions.
For this reason we interpret the geometrical form of the six available musical chords, expanding the original tonnetz grid to obtain an inner empty zone, which can allow the transition to the other remaining five chords. The proposed geometrical representation, depicted in Figure 6, is a circular ring sliced in six parts corresponding to the six chords. To clarify the relationship between this geometrical form and the tonnetz, we superimpose the tonnetz chord displacement on the upper part of the ring. The bigger thin lined triangles are the primary chords, while the squeezed ones are the parallel chords. To obtain the required linking space, we project the parallel chords in the lower part of the ring. The ii degree is overturned along the F-D axis and is placed just under the IV (following the dashed arrows). The iii degree is overturned along the E-B axis and is placed just under the V degree. The vi degree is overturned along the A-E axis and is placed in front of the I degree. This new displacement provides a logical chord disposition, where from every chord it is possible to reach each of the remaining five chords without touching the others.
Summarizing, the geometrical interpretation of the outstanding tonal melodies features to be used in the Harmonic Walk application is a straight line and a circular ring.
5. System Architecture
In this section we describe the Harmonic Walk architecture which is composed of two software modules, aimed at video analysis and sound production, respectively.
In Section 5.1 we describe the Zone Tracker application with its video analysis algorithms and masks for surface division. In Section 5.2 also the Max/MSP6 patch, responsible for the audio output, is presented. The two software modules communicate through the OSC protocol7. The application’s interface is explained in Section 5.3, where the spatial displacement of the musical features’ geometrical interpretation is introduced.
5.1. The Zone Tracker Application
A ceiling mounted video camera, oriented perpendicularly to the floor, captures the users movements inside a rectangular area, whose dimensions depend on both the distance from camera to floor and the field of view of the lens (see Figure 7). As the system is designed for carrying out educational activities inside a classroom, the camera lens is chosen in order to view a rectangle of about 3 × 4 meters when the camera is mounted on a 2.8/3-meter-high ceiling. The video application, called Zone Tracker, tracks the user’s position employing a series of masks. These subdivide the active surface, allowing detecting the portion of space occupied by the user. The video analysis algorithm analyzes the input images in three steps, as shown in Figure 8.
(1) Background Subtraction. The background is subtracted following the averaging background method proposed by Jabri et al. [21]. The method used is based on statistical modeling of the background for each channel color and detection with adaptive threshold. The background model is obtained by calculating for each channel the average and the standard deviation of each pixel in the course of a certain number of frames acquired from static shots of the background, following the formula:where is the average calculated up to frame , is the learning coefficient of the model and is the pixel value at the frame . After building a statistical model of the background, the new frames are compared with it, thus producing what is called a confiance image, that is, a grayscale image, where the value of each pixel represents the probability that it forms part of a region of the image not belonging to the background.
(2) Morphological Image Processing. These images are processed by means of morphological transformations [22]. Dilation and erosion are the two basic mathematical operations which correspond, respectively, to the expansion and thinning of the black region. A combination of the two leads to opening and closure of morphological operations, aiming at defining well shaped silhouettes blobs. When the value of confiance does not allow deciding for sure whether a region is part of the figure of interest or background, a threshold algorithm can be used. This threshold function was implemented through an operation of nonlinear convolution between the confiance image and a mask (kernel), following the formula: where represents a pixel of the source image and a pixel of the image to which the transformation was applied. An uncertain pixel (with nonzero value) is not part of the background if there is at least one pixel equal to 1 in a neighborhood .
(3) Blob Tracking. The blobs moves are tracked and the two-dimensional barycenter of each blob is calculated. For this purpose, the OpenCV addon on openFrameworks provides the classes ofxCvContourFinder and ofxCvBlob. In particular, the findContours method in ofxCvContourFinder, given an input binary image, employs some features of OpenCV to detect blob and store the information in data structures of type ofxCvBlob. Although the video analysis module can track more users simultaneously, this first implementation of the Harmonic Walk is designed to be used by a single user at a time.
5.2. The Zone Tracker’s Masks and the Audio Rendering
The users barycenter coordinates are employed by the Zone Tracker application to compute the occupied zone depending on the different masks stored in the application’s memory. In this implementation of the Harmonic Walk, we use two different masks corresponding to the two geometric representations discussed in Section 4. The first mask subdivides the tracked zone in 25 squares, whose dimensions, after empirical observations, have been found to correspond approximately to the distance of a human step. This mask is employed to project the straight path of the musical unit sequence. The circular ring mask is subdivided into six zones. The radius of the outer and inner circles is defined accordingly to the same principles as the dimension of the squares. This mask is employed to project the circle of the six chords of the harmonic space. Zone Tracker sends via OSC the numbers corresponding to the various regions to the Max/MSP patch. Here, for each tonal composition two kinds of audio files are stored: one kind is the music audio file segmented following the principles illustrated in Section 3.1; the second kind is a group of audio files which reproduce the chords which belong to the song’s key chord space and among which there are the chords employed in the song. The chords are played using a MIDI synthesizer with the same rhythm and timbre of the original composition.
5.3. The Harmonic Walk Interactive Interface
The Harmonic Walk user interface consists of a rectangular area whose dimension changes depending on the ceiling height. To help the user’s perception of the active area, we put the straight line of the musical unit sequence along the borders, while we put the center of the circular ring at the center of the rectangle8.
The user’s paths are identified through visual tags which are positioned at the center of each zone. There are two employed paths: the first is the one corresponding to the straight line and is marked with the white crosses, while the second is the circular one, marked with the black crosses (see Figure 9). The beginning of each path is marked with an arrow. In this way the user has a visual cue of the center of the various regions and is guided through them by his inner feeling of the music structure9.
6. The Harmonic Walk’s Assessment
The Harmonic Walk’s assessment tests were organized at the Catholic Institute “Barbarigo” in Padova during the first decade of December 2014. We wanted to gauge at what extent the application’s design is successful in introducing the user to the tonal music features without any previous information about the application’s task and materials. To do so, we quoted the subjects’ success in the melody segmentation recognition and in the melody harmonization tasks. Moreover, to weight the manifold musical contents of the Harmonic Walk, we prepared a questionnaire to be fulfilled by the subjects (see Section 6.5) for both quantitative and qualitative assessments. In particular, we were interested in evaluating if, after the experience of the test, the subjects could gain some musical knowledge and consciousness about the musical features employed.
6.1. Subjects
A total number of 22 high school students between 16 and 20 years old took part in the test. The students were equally subdivided into two different groups: one musically trained and the other not. The first group was taken from classes belonging to the music high school, with specific instrumental and music theory programs. The second was chosen from classes belonging to various kinds of high schools with no music programs in their curricula. Among these we asked for students with no musical practice at all. For preserving clarity in the test’s results, we excluded from the test only subjects with a declared state of disability.
6.2. Materials
The choice of the music genre and of the music piece which could fit better our student’s sample was not trivial. We choose the song as one of the most popular genres of music. Moreover the sung words can help learning, memorization, localization of melodic qualities, and performance. Then we decided to bypass the most up-to-date young people major hits for at least two reasons: these songs’ sample could be too restricted for our needs (i.e., it could not offer enough options regarding chords complexity, harmonic rhythm, and so on); moreover, as not all young people listen to the same kind of music, the choice could not guarantee that all the students know the same song.
So, we started with defining the ideal features of our song:(i)The chords employed must belong all to the same key, to fit the six roots’ harmonic space explained in Section 3.2.(ii)The song’s harmonic progression should include, beyond two or three primary chords, at least one, but not more than two, parallel chords, to offer a medium level of harmonic complexity.(iii)The chord’s change (i.e., the harmonic rhythm) must be not too fast to allow easy body movements in the physical space.(iv)The song must be very popular, clear, and easy with the melodic structure.After an accurate search, we choose Adriano Celentano’s Il ragazzo della via Gluck10, whose musical structure is summarized in Table 2. The first musical phrase of the song is composed of 11 segments built upon 3 chords (two primary and one parallel). The column of the “syllables of change” shows the syllables of the text corresponding to the harmonic changes, that is, the points where the melodic segmentation occurs. The table shows that the points of segmentation are 10, as well as the number of chord changes. The column “Duration” outlines the number of bars occupied by the different song’s units. As can be seen, the harmonic rhythm is not constant; that is, the song’s units do not all have the same duration.
6.3. Scenario
In Section 4 we discussed about two musical features (melodic segmentation and harmonic space), with the two related geometrical interpretations and masks (the straight line and the circular ring). At the present research stage, for the Harmonic Walk’s test, we planned a three-phase approach to the tonal melody accompaniment, as showed in Figure 10.
In the first phase the user is presented to the straight path along the borders of the active area, corresponding to the source-path-goal schema shown in Figure 5. Every step through the mapped regions produces the sound of the portion of the audio file corresponding to the segmentation described in Section 3.1. If the user fails to step to the next zone in time (i.e., if s/he is not aware of the song’s harmonic rhythm), the audio file ends and the performance is interrupted. In the second phase the user enters the circular ring containing the six roots discussed in Section 3.2, with the aim to explore the harmonic space and to couple the spatial position to the chords sound and function. In the third and last phase, starting from the melodic segmentation knowledge acquired in the first phase and from the experience of the harmonic space acquired in the second phase, the user can now try to find the chords useful for the melody accompaniment of the composition. As usually there is not a unique solution to the melody harmonization, the user can experiment various chord sequences and choose the one s/he prefers. We consider the experience complete and successful as soon as the user harmonizes the whole given melodic excerpt without interruptions and in a way s/he considers valid.
6.4. Method
The high school students were tested individually in private sessions where only the test conductor and the music teacher were present. No previous information about the song used during the test was provided to the subjects. Each student was presented with some written instructions about the tasks s/he had to accomplish, while a short demonstration about the environment and its interaction modalities was provided by the test conductor. After this, the student undergoes the first phase of the test, during which s/he listens to the song, trying to understand where the units sectioning points are and trying to link them through an appropriate pace timing. The student is given a maximum of 5 minutes’ time to conclude the first phase of the test. In the second phase the student enters the circle of the song’s harmonic space and explores it in a free way. S/he has 3 minutes to listen to the chords and to remember their position in space. In the third phase the student is given the text of the song and, with the help of the teacher who sings the melody, tries to find the chords which can fit the song. As soon as s/he feels that the chord is not correct, the song begins again and continues till the end or till the next mistake. The student is given 5 minutes to complete this task. After the time expires or the last task is accomplished, the student is presented to a questionnaire for both quantitative and qualitative feedback about the test. The test conductor fulfills also a form of qualitative assessment and further observations. Moreover, for each of the three phases of the test, the system records the hit zones and the overall time employed by the subject.
6.5. The Questionnaire
The questionnaire prepared for the test subjects contains open-ended interviews, true-false and multiple choice questions, Likert-scale surveys11, and participant observations, as shown in Table 3. It focuses on the first and third phase of the test and it is designed to collect both quantitative and qualitative data from the test subjects.
For the first phase, we prepared an interview about the detection of the song’s syllables where a harmonic change occurs, a true-false question about the duration of the song’s units, and a multiple choice question about the principle upon which the unit’s had been determined and asked for an opinion about the difficulty of linking correctly the various song’s units.
We put no question about the second phase of the test, whose data are collected through direct observation and subjects’ movement logs.
For the third phase, we put two open-ended interviews about how many times the subject did a chord change and how many overall chords s/he used. After this, we asked to draw on a plotted schema of the six chord circular ring, the route the subject did for harmonizing the melody.
At last, to collect the subjects’ application’s evaluation, we propose two Likert-scale surveys about the test experience’s pleasantness and utility. Both surveys have 5 items. The first survey includes the following items: Very disagreeable (1). Disagreeable (2). Indifferent (3). Pleasant (4). Very Pleasant (5).The second survey includes the following items: Incomprehensible (1). Useless (2). Indifferent (3). Useful (4). Very Useful (5).As this is the first rigorous assessment test we conduct for the Harmonic Walk application, we could not base the item’s definition upon previously collected users’ statements. So, we opted for rather neutral items. The questionnaire ends with two opinion questions about the subjects’ feeling and musical principles employed in the test.
6.6. Results and Discussion
In this subsection we gather information obtained by questionnaires, direct observations, and movement logs and organize our results in quantitative (Section 6.6.1) and qualitative assessment (Section 6.6.2).
To help fluency and clarity of reading, we discuss the obtained results from time to time, summarizing them and adding any further consideration in Section 6.6.3.
6.6.1. Quantitative Assessment
The aim of the Harmonic Walk quantitative assessment is firstly to verify at what extent subjects have been successful in identifying the harmonic changes and in harmonizing the melody. Secondly, we gather and discuss the information obtained during the exploratory second phase of the test. The analysis of the subjects movements and behavior is not only quantitative (the logs recorded by the system during the test), but also qualitative (the test conductor observations); nevertheless, we report them together. Thirdly, we want to quote how much musical knowledge has been acquired by the tested subjects during the experience with the Harmonic Walk and what the tested subjects think of the experience with the application.
The harmonic changes perception is the first cognitive result we expect after the test proposed in the first phase. The user had to wait the end of the song’s harmonic unit before linking the next with his step. In this way s/he should have reinforced his/her perception of the harmonic changes. To verify if this knowledge has actually been realized by the test participants, we ask them to underline the song syllables where they felt a harmony change occurred. The answers results are shown in the column “Syllables” of Table 7.
As expected, the results of nonmusicians are beyond those of musicians. The best record of hits over 10 syllables to detect is 6 for nonmusicians and 9 for musicians. The worst is 0 for nonmusicians and 2 for musicians. The hits’ average changes significantly also depending on the song’s harmonic unit as can be seen in Figure 11. As a matter of fact the hits’ average drops dramatically among nonmusicians (but also, less dramatically, among musicians) as soon as the song enters the last 4 harmonic changes, those involving the parallel harmony (vi degree), while it is rather high at the beginning of the song.
The melody harmonization is the conclusive main task of the third phase of the application’s test. The test conductor reports a complete successful harmonization when the subject can find the chord sequence that s/he considers a good harmonic solution for the song and that s/he is able to repeat. No one of the complete or partial successful harmonization differs from the model proposed in the original song. Again, the results show the difficulty in the perception of the minor parallel harmonic areas. As can be seen in Table 4, among the 6 harmonization only 2 were complete, while the other 4 could accompany the song only in its simplest part, the one involving the tonic and dominant chords.
Moreover, we asked the subjects to draw the path they performed during the harmonization test on the circular ring, marking it with arrows or numbers. We divide the records in groups depending on the number of right chord sequences identified by the subjects. The results are shown in Table 5.
In the first row we reported the number of not answered questions and in the second row the number of answers where only the first chord was right and the second was wrong. This means that the subjects failed to identify even the first chord progression of the song, which was instead identified by the subjects of the third row. The fourth-row subjects show to have understood that the sequence returned on the first degree at least once, while the fifths row subjects caught that the sequence was repeated more than once. At least, identifying more than 6 right chords means that the subject has found also the vi degree progression. Nonetheless, there is never a full correspondence between the right or partially right melody harmonization subject and the identification of the right sequence as can be seen in Table 6. Whereas the correct chord sequence isin Table 6 we see that in all the sequences there is always some mistake, or any missing chord (like in subject number 6 where there are only the last two chords missing).
In the phase of the harmonic space exploration the subject has the possibility to explore the circular ring where the chords are laid. This experience is crucial for the success of the next required task, the melody harmonization, because it is here where the subject builds her/his spatial map of the harmonic relationships. Therefore, it is important to analyze what kind of movements subjects did while exploring the circular ring. From direct observation and from the analysis of movements recorded by the system during the test, two different approaches to the circular ring exploration were found. In the first approach the subject follows the shape of the circle starting from the left or from the right side. S/he touches all the six zones in sequence, obtaining the following harmonic progressions: which are very long progressions without the repetition of the tonic chord. The latter is a fundamental perceptual anchorage point for tonal progressions and if it fails to appear for a certain time, the harmonic progressions lose their direction. Thus the experience does not reinforce tonal chords localization as expected.
The second approach emphasizes movements where the frequent return to the tonic chord is present. This exploration approach is nearer to the actual tonal composition’s structure, where the tonic frequently occurs and where chordal progressions are often repeated. Returning to the tonic chord helps to distinguish harmonic functions; furthermore, the repetition of the movement among two well identified space regions trains the body to perform simple and strong harmonic progressions. As demonstrated by the recorded paths (see some examples in Figure 12) and by direct observation of the subjects’ movements, the exploratory phase has been only seldom characterized by recursive movements on the tonic chord. Sometimes we registered and observed such movements like 1-6-1 (tonic, dominant, and tonic) or 1-2-1 (tonic, subdominant, and tonic), but never movements like 1-4-1 (tonic, minor parallel, and tonic) or 1-5-1 (tonic, mediant, and tonic), nor perceptually strongest progressions like 1-2-6-1 (tonic, subdominant, dominant, and tonic). The exploratory phase was apparently left free of any biasing element, in the sense that no hint was given to the subjects about the task to accomplish. It seems that, in such conditions, the shape of the projection (the circular ring) has been the most important cue to follow for the subject. Indeed, in nearly all the recorded paths we found at least one circular path while in not all the recorded paths we found the recursive path. Moreover, many of the subjects could not fully understand the difference between the straight path and the circular one and, consequently, showed a clear tendency in using them in the same way, that is, following the shape’s contour. On the other hand, the straight logic of this behavior must be recognized and seriously considered in the future work of refinement of the application design. As already outlined, when following passively the ring, many subjects missed the possibility of building a perceptual and a sensorimotor map of the strongest tonality harmonic progressions, thus compromising the possibility of success in the third phase of the test. The search of a satisfactory harmonization path has nothing to do with a random search but follows very clear, perceptually strong, recursive tracks. Of course, we did not expect to find test subjects perfectly aware of these musical structures, but, rather, our aim was to encourage them to discover such features through a perceptual and movement practice.
(a)
(b)
(c)
To find out if the tested subjects have gained some knowledge about the song they worked on during the test, we asked them some questions about the musical features employed for accomplishing the test’s tasks. The first question concerned the song’s harmonic rhythm, and particularly if the song’s harmonic units had all the same duration. Nearly the 100% of the subjects answered correctly (i.e., that the song’s units do not all have the same duration as showed in the last column of Table 2).
The second question was about the principle upon which the units had been determined. The great majority of subjects (9 nonmusicians and 10 musicians) checked the chord change against rhythm repetition and melody repetition, showing thus to have understood the segmentation principle.
Other less simple questions were about how many times the subject changed the chord and how many different chords s/he employed during the third phase of the test (the melody harmonization), respectively, column of “Changes” and of “Emp. ch.” (number of employed chords) in Table 7.
The interpretation of these results is not immediate and can be discussed mainly when they are compared with the results of the first and third phase of the test. For instance, the question on the number of chord changes (column of “Syllables”) is directly related to the detection of the syllables of harmonic changes (column of “Changes” in Table 7), as the number of changes is exactly corresponding to the number of syllables detected. Basically the question is the same but is put in a different perspective. It seems that the subjects did not realize the similarity of the two questions, as the answers from the same subject differ in all the cases but one.
Only subject number 7 among the musicians group gave both correct answers, while subject number 9 gave the most coherent answer having detected only 4 syllables of change but also 4 chord changes. Also, if we compare the records of the successful or partially successful subjects in the melody harmonization task (starred and double starred subjects) we see that the correlation is scarce or null. Only subject number 2 among the group of musicians achieved a good record in the number of chord changes, while all the others made a number more or less great of mistakes. Similarly, comparing the number of different employed chords (3 in total, I, V, and vi degree) we see that there is a good match only with subjects 5 and 6 of the musicians group, while the other four starred subjects declared all the wrong number of chords.
In the questionnaire we prepared two Likert-scale surveys. In the first survey we asked to express a judgement about the application’s pleasantness and in the second we asked about the utility of the experience made during the test. The results are, respectively, shown in Figures 13 and 14 and discussed in Section 6.6.3.
6.6.2. Qualitative Assessment
Data about qualitative assessment come from the three participants’ opinion questions contained in the questionnaire. The first opinion question concerned the difficulty the subjects found in linking the different song’s units in the test first phase, the second the subject’s perception of the application, and the third what musical principles they thought were involved in the test.
Four nonmusicians subjects and 2 musicians reported difficulties in understanding the right moment when it was necessary to move on to change the song’s unit. Three subjects from the nonmusicians group reported difficulties in understanding the harmonic structure of the song, pointing out so that the question had not completely been understood but rather that there was some confusion about the first-phase and the third-phase tasks. Only one subject blamed the lack of previous knowledge of the song used in the test.
Some subjects judgements, ideas, impressions, and feelings about the application and its use are partially reported in Table 8.
Other judgements, expressed only by one subject, were strange, incomprehensible, frustrating, complicated, indifferent, engaging, nice, and beautiful. It is also interesting to look at the reasons why such judgements were expressed. The application was considered “useful” by nonmusicians:(i)To develop the memory of listening.(ii)To evaluate the understanding of musical features.(iii)To understand the musical experience.And the application was considered “useful” by musicians: (i)To keep in time while playing with others.(ii)To work with children.(iii)To keep coordination and precision in music performance.(iv)To learn harmony changes not in theory, but in practice.(v)To develop aural skills.(vi)To test your own musical skills.
It was considered “interesting” by nonmusicians:(i)For discovering how complex a popular song may be.(ii)For seeing how the body reacts to music. And it was considered “beautiful” by 1 musician subject because it can introduce people to music. As can be seen, subjects reported a great number of cues and a variety of ideas about the experience during the test.
We asked the test subjects to point out the main principles upon which the application was designed.
As can be seen in Table 9, musical memory and rhythmic skills are the strongest perceived musical features employed in the design of the Harmonic Walk.
6.6.3. Summary of Results
To make a rigorous test of the Harmonic Walk application, we set very hard and strict assessment conditions:(i)High level of musical features involved in the application design.(ii)No previous hints about the application cognitive contents, nor about the song employed during the test.(iii)Strict test protocol and time constraints.
The recognition of the syllables where the song changed its harmonies and where the subject had to make a step to continue the song performance hits an average of 3.6 for nonmusicians and of 4.8 for musicians, over 10 syllables. The average is rather low, but we saw from Figure 11 that it decreases a lot depending on the song’s harmonic area (minor parallel area), demonstrating a difficulty in parallel harmonies perception. Nonetheless, as this area arrives in the second part of the song excerpt, the reason of the drop of the number of detected syllables could be also a lack of memorization. The full harmonization task was accomplished only by 1 nonmusician and by 1 musician subject. The other 4 musician subjects accomplished the task only in the first part of the song, involving only the tonic and the dominant chord. So, among 22 tested subjects we recorded a complete and partially successful accomplishment task for 6 subjects, a rate of 26% of the total of the sample.
Nearly the complete sample of nonmusicians and musicians subjects at the end of the test was conscious that the song’s harmonic units do not all have the same duration and that the principle upon which the song’s unit had been determined was the chord change. But when asked how many chord changes had occurred in the song excerpt, the number of changes drops at 4.8 for musicians and at 7.5 for nonmusicians, over 10 chord changes, with 3 correct answers. The question about how many overall chords were employed in the song (3) shows an average of 3.5 among nonmusicians with 1 correct answer and an average of 4.2 among musicians, with 2 correct answers. This question is crucial for the tonal language musical knowledge, because it shows the difference between the chord changes (10 in eleven harmonic units in the straight path) and the number of places to reach in the circular ring (3, corresponding to the tonic, dominant, and minor parallel chord). This difference is exactly the effect of the recurrence of the tonic-dominant and tonic-minor parallel progressions. In general, we observed a scarce correlation between the partial or total success of the subjects in the harmonization task and the correctness of their answers in the questionnaire, as can be seen in Table 10, where the asterisks outline the correct answers. The explanation of such results is not simple. The results of the column of the syllables of change, mainly for the complete successful subjects, are particularly touching and need further investigations. How can a successful harmonizer miss detecting the syllables of the song where he actually searched and successfully found the right chord? It seems that consciousness lays far away from the embodied knowledge employed by these successful subjects. On the other hand, subject number 7 of the group of musicians detects perfectly all the syllables of change (10) and the number of harmonic changes (10) but misses finding the number of employed chords (5 instead of 3); nevertheless he fails to harmonize the melody in the given time.
The psychological and cognitive weight of the test’s experience is witnessed by the rich and complex output we obtained from the participants. It seems that, independently of the explicit knowledge they succeed to realize at the end of the experience, the Harmonic Walk is able to offer to the users a deeper insight of what a musical experience is. The application’s use is not fatigue, as the average of ratings about its pleasantness is 3.15, while the average of ratings about its utility is 3.25, both a little above “indifferent” (see the pleasantness and utility item list in 6.5). In general, the main perceived features upon which the application is built are musical memory, rhythmic skills, and sensorimotor reactions. Not to be ignored, a total of 3 subjects (13% of the sample) labeled the application as “difficult.” This result can be interpreted as the other side of the coin of richness of musical contents, because, in spite of its simple interaction modality, the application actually conveys in short a lot of information about complex compositional processes. On the other hand, this is just the field where a proper didactic approach and careful training can help. Moreover, most insightful address to the Harmonic Walk could also highlight the great potential of creative use, which is implicit in its design and which nobody among the tested subjects did even mention.
In general, in the quantitative assessment’s records, musicians obtained better results than nonmusicians, as can be seen at glance in Table 7 and as reported in Table 11.
The number of successful harmonizers was 1 among musicians as well as among nonmusicians, while no partially successful harmonizer was found among nonmusicians. If we look at Table 5 we see that musicians were much better than nonmusicians in identifying the right song chord sequence, as the majority of them (54%) could identify a right sequence of more than three chords, while all the sample of nonmusicians equals or remains above this number. In general, in Likert-scale surveys, musicians quoted better the application’s utility while nonmusicians quoted better its pleasantness. In particular, in the application’s perception opinion question, 2 nonmusicians judged it as “useless,” showing that the mere test’s experience cannot be considered self-explanatory for all the subjects.
7. Conclusions
In this paper we introduced the Harmonic Walk application, designed for practicing and studying the tonal melody accompaniment. To verify the application’s utility as a didactic tool, we organized an assessment test involving 22 high school students, subdivided into two groups: one from a music high school and the other from other high schools. This second group was composed of subjects with no musical practice at all. Below we summarize the most important results:(1)The 26% of the tested subjects could accomplish the final application’s task: the melody harmonization. Among these only the 9% could harmonize the complete song, while the remaining 17% harmonized only the song’s first part, characterized by a very simple chord progression.(2)The harmonic changes perception, expressed by the detection of the song’s syllables of change, hits an average of 3.6 detected syllables for nonmusicians and of 4.8 for musicians over 10. The number of detected syllables decreases in the second part of the song’s excerpt, characterized by a parallel chord progression.(3)Some musical knowledge about the employed musical features is confirmed by the assessment results, but the consciousness of musical concepts does not coincide with the subjects who actually accomplished the application’s main task.(4)In the qualitative part of the assessment, we got a very rich and various feedback from the test subjects, who in general reported a lot of interesting musical cues from the experience.
So, we can conclude that the Harmonic Walk is a strong, enactive tool which can provide a fully immersive and embodied experience of complex and fundamental musical concepts. It can convey a lot of implicit information about the employed musical features which can be considered as a rich basis for music learning programs, whose aim is to help all the implicit knowledge to emerge. Its use is simple and not uncomfortable for the user. Conversely, it has been considered useful, amusing, and interesting by the majority of the test subjects.
7.1. Further Work
The future development of the Harmonic Walk concerns essentially two domains. The first is the music didactics domain where a lot of work is necessary to build training programs to work with the students in the application’s physical interactive environment. After having tested the great potentialities of the Harmonic Walk in the music field, it is important to focus on some critical cues emerging during the test:(i)The difficulty in the perception of the parallel chords, reported more than once when discussing the results(ii)The confusion aroused by the different use of the two geometrical projections: the straight line, which had to be followed step by step, and the circular ring which instead had to be explored and used employing some recursive harmonic pattern. It is clear that something must be done to suggest to the user that s/he has to change the interaction modality when entering the circular ring mask. Nonetheless, we do not believe that corpus-based statistical tools algorithms, already employed in other similar applications, will help, as the Harmonic Walk’s aim is not to give to the user information about the melody accompaniment solution, but to help them to find their solution through a perceptual exploration based on body movements.
The second domain is the study of the application’s features from the point of view of its novelty in the human-computer interaction field. The Harmonic Walk, as a motion-based application, belongs to a new generation of human-computer interfaces where the interaction style evolves toward the physical world rather than remaining confined in the computer screen [23]. This means that the user’s environment awareness and body skills become the basis of the interaction with the digital world, bringing with them all their expressive power and efficiency. Considering the application from this point of view, other computational, cognitive, and psychological cues emerge, like implicit and tacit knowledge [24], the extended mind [25], and situated cognition [26].
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The authors want to thank Leonardo Amico for his support in programming the Zone Tracker application, Cesare Contarini, Dean of the Catholic Institute “Barbarigo” of Padova, and, particularly, Giuseppe Viaro, Music Technology Teacher, for their cooperation during the assessment tests.
Endnotes
- A video showing some tests with the Harmonic Walk can be found at https://www.youtube.com/watch?v=OjwXfzq_CkU.
- Literally “web of tones”: the tonnetz appeared for the first time in a harmony treatise written by Euler in 1739. In the 19th century the tonnetz was reviewed and discussed by the German music theorist Hugo Riemann.
- With respect to the fundamental, the first interval introducing a new pitch is the fifth, followed by the major and the minor third. A chain for each of these intervals is spread on the three axes of the tonnetz, where each triangular area represents the consonant major and minor triads.
- With respect to the fundamental, the first interval introducing a new pitch is the fifth, followed by the major and the minor third. A chain for each of these intervals is spread on the three axes of the tonnetz, where each triangular area represents the consonant major and minor triads.
- After George Lakoff’s and Mark Johnson’s theory, we can understand abstract concepts through image schemas which are stored in our body cognition and which come from bodily interactions.
- Max/MSP (https://cycling74.com/) is a visual programming language for audio and video production, algorithmic composition, and signal processing, written by Miller Puckette in 1980.
- OSC, acronym of Open Sound Control [28], is a communication protocol based on modern network technology (http://opensoundcontrol.org/introduction-osc) which was developed and continues to be a subject of ongoing research at UC Berkeley Center for New Music and Audio Technology (CNMAT). Originally intended for sharing music performance data, it is commonly used to exchange information between computers, applications, and various multimedia devices.
- We represent the masks as square and circular ring also if, due to camera pixel shape, the actual masks are, respectively, distorted in a rectangle and in an oval ring.
- The tags both in the straight line and in the circular ring are not descriptive; that is, they are not marked with the syllables of change or with the chord names, but they have the function to indicate to the user the right position to be occupied on the application’s surface.
- Celentano [29] is a popular Italian singer, very known also abroad. He has begun his career in the sixties. Il ragazzo della via Gluck, written in 1966, was one of his greatest hits.
- The Likert-scale is a technique used to scale responses in survey research. It employs at least 5 items ranging from negative to positive statements, allowing so an easy and fast computation of the test’s results.