A Combined Mathematical Treatment for a Special Automatic Music Transcription System

Abstract

This paper presents a combined mathematical treatment for a special automatic music transcription system. This system is specially made for computer-synthesized music. The combined mathematical treatment includes harmonic selection, matrix analysis, and probability analysis method. The algorithm reduces dimension by PCA and selects candidates first by human auditory model and harmonic structures of notes. It changes the multiple-F0 estimation question into a mathematical problem and solves it in a mathematical way. It can be shown in this paper that the experimental results indicate that this method has very good recognition results.

1. Introduction

Music transcription means an act of listening to a piece of music and writing down music score for the piece. The traditional way of making music is just like that: a performer reading a score, playing an instrument, and thus producing music. Transcription of polyphonic music is a reverse process: an acoustical waveform is converted into a parametric representation (such as MIDI), where notes, their pitches, starting times, and durations are extracted from the signal [1]. Automatic music transcription means converting acoustic music signals to musical scores automatically by computer analysis technology. Automatic music transcription greatly reduces the manual labor and time and it becomes a key technology in music signal processing [2, 3]. Automatic music transcription can be widely used for content-based music retrieval, low-rate coding, automatic musical accompaniment system, and so on.

Fundamental frequency (F0) is an essential descriptor of harmonic sound signals such as speech and music, and it determines the pitch of a music note. Single-F0 estimation algorithms assume that there is at most one musical of which the F0 is to be extracted. Although single-F0 estimation algorithms have been considerably developed, their applications to music signals are somehow limited because most music signals contain more than one concurrent harmonic source. Multiple-F0 estimation algorithms are thus required for the general case and it needs to estimate each of the sources which play together, which is one of the most important parts of the automatic music transcription [4, 5].

The most important contribution of this paper is solving the multiple-F0 estimation in a mathematical analysis method. The signals we need to process are wave forms in time domain x[m]. At first we need to change it into frequency domain by 𝑋(𝑛,𝑘)=^∞_𝑚=−∞𝑥[𝑚]𝑤[𝑛−𝑚]⋅𝑒^{−𝑗(2𝜋/𝑁)𝑘𝑚} (1.1) or cepstrum domain by 𝑥_𝑐[𝑛]=12𝜋^𝜋_−𝜋log𝑋(𝜔)𝑒^𝑗𝜔𝑛𝑑𝜔. (1.2)

And then we change the multiple-F0 estimation question into matrix linear equations. It is an ill-posed equation, so we can solve it using a truncation completely least squares method. Based on this, we can use a priori probability to filer the result to improve the correct rate.

This paper is organized as follows. At first it introduces the research background and situation of study of automatic music transcription and multiple-F0 estimation. And then it describs our algorithm and experimental stages in detail. We focused on the effect of harmonic structure in primary selection progress and the math model of multiple-F0 estimation. And it is followed by an experimental results display, and finally there are the conclusion and perspective and acknowledgment.

2. Automatic Music Transcription System

The system architecture of automatic music transcription system proposed in this paper can be found in another paper by the same author in [6]. In this system, input music signal passes through a spectral analysis module first and then performs a multiple-F0 estimation progress after preprocess and windowed DFT. The following multiple-F0 estimation algorithm is performed for each music frame to predict one or more notes that appear in this time slice. In the end, this algorithm will detect the starting time and the ending time of each note. Through these steps, a music signal can be transcribed into note event sequence, which can be described in MIDI form or in music score directly.

The key process of this system is multiple-F0 estimation and it is also the main contribution of this paper. The multiple-F0 estimation algorithm has two main stages: training and identify. It contains three main steps in training process: preprocess, PCA, and signal normalization, and six main steps: enframe, preprocess, PCA, signal normalization, frequency primary election, and multiple-F0 estimation. The steps which have the same name did the same things.(i)Preprocessing: it is the first step in training process. It removes the silent part in the front and back of the notes section first and then through a windowed DFT transformation. (ii)Normalization: signal normalization means converting the signal after preprocessing into equivalent one with mean 0 and variance 1, and it is used to facilitate the follow-up treatment. The signal is to be separated into two parts after normalization. One will go into PCA step, and the other will be used to analyze the harmonic structure. (iii)PCA: PCA means principal component analysis. It makes all the notes together by a common preextraction of principal components. That is to map all the notes from a higher-dimensional space to a relatively low-dimensional space coordinate system. While PCA is used to train date here, we can define a PCA score (𝐼_pca), which means that we only retain the components whose sum of ratios is greater than the score.(iv)Calculate 𝐻_𝑛: in this step we need to decide the number of harmonics for each note and the amplitude of spectrum for each harmonic. In fact, each note has clear harmonic characteristics. It means that there will be clear impulse near integer multiples of the fundamental frequency. The same notes played by different instruments have harmonics at almost the same location but the margins of each note may be different. Using this feature, we can roughly estimate a certain audio whether contains a note. This feature can be used to select candidates for the fundamental frequency to facilitate follow-up treatment.(v)Enframe: the length of the music to be detected is variable; thus, an enframe processing is needed in order to obtain a more accurate detection and estimation of the notes. Each frame will have a detect result, which shows the notes that the frame included.(vi)Frequency primary selection: in order to reduce the computation and improve the estimation accuracy of data, before the multiple fundamental frequency estimation step, a frequency primary selection step needs to be produced. In this step, all candidate notes are estimated. After this, it only needs to select the final notes from these candidate ones instead of from all the notes, which can greatly benefit the speed and estimation accuracy. The detail information will be described in the next section. (vii)Multiple-F0 estimation: the multiple fundamental frequency estimation here means to estimate the consisting of notes from candidates based on the data after the above processing. This is a core part of the whole system and we can use some mathematical methods to solve this problem. The detailed information will be described in the next section.

3. Using Harmonic Structure

3.1. Essential Knowledge

The timbre refers to the feeling features of the sound, such as frequency, loudness, and the duration. The timbre has an identity function, and we can distinguish two kinds of sound with the same pitch and intensity but coming from different musical instruments according to differences in sound timbre. The timbre is a multidimensional object. In addition to frequency, loudness, and duration, it also contains amplitude envelope and spectral envelope.

Spectral envelope is an important parameter to describe the sound in frequency domain and it is constituted with the amplitude of all harmonics. We can describe the spectral envelope by harmonic structure, labeled as 𝐻_𝑛: 𝐻_𝑛=𝑎₁,𝑎₂,…,𝑎_𝑛, (3.1) where 𝑎_𝑘 is the amplitude of the kth harmonic and 𝑛 is the number of harmonics.

In accordance with western musical convention, note events are ordered using a logarithmic scale [7]. For linear frequency to MIDI conversion, the following expression can be used: 𝑛=69+12log₂𝑓₀.440 (3.2) In music notation, each note is named with one of the following symbols: Do Re Mi Fa Sol La Ci.

A sequence of notes fromDo to Ci is called an octave. In a given octave, the fundamental frequency of each note is an integer multiple of fundamental frequency of namesake from previous octaves. Since the harmonics of each note are also integer multiples of fundamental frequency, these harmonics represent namesake notes of it in next octaves [8]. For example, the fundamental frequency of La in octave 4 (i.e., La4) is 220 (Hz). So, the frequency of its second harmonic is 440 (Hz) that is equal to the fundamental frequency of La in octave 5 (i.e., La5). This is the frequency overlapping problem which we mentioned above and it is also a key problem when performing the iterative deletion.

Spectral envelopes of different instruments have obvious differences, while the same kind of instrument has similar spectral envelope. Music played by the same instrument has a high degree of similarity and a stable harmonic structure. However, the computer-synthesized music is established by the same soft wavetable. So we can believe that the harmonic structure of computer music synthesized by the similar instrument is almost the same. This paper assumes that the harmonic structure is unchanged when the F0 of one note is changing weakly in a semitone.

3.2. Usage of Harmonic Structure

Based on the above characteristics, the information of harmonic structure can be used to improve the multiple fundamental frequency estimation algorithms. In our algorithm, the information of harmonic structure is mainly used in two places: in the training stage, obtaining the information of harmonic structure of each note played by different instrument, and in the identification stage, using the harmonic structure matching rate to determine the candidate fundamental frequency, to increase the accuracy of following multiple-F0 estimation.

In the training stage, the parameters of each note can be extracted from the training materials, each training material contains only one note, and each note can be trained by 100 materials. We analyze the spectral envelope of each material and calculate the harmonic structure 𝐻_𝑛. We set a training threshold th1. If the normalized spectral envelope is larger than th1, the corresponding element in harmonic structure was set as the harmonic amplitude, otherwise was set to 0.

For an in-harmonic instrument, partial frequencies 𝐻_𝑛 can further deviate from this frequency range. As a consequence, these partials are missed or assigned erroneously to other partials by our system [9]. However, this situation only occurs for strongly in-harmonic instruments and at high values of parameter. Partial frequencies for an in-harmonic instrument can be expressed by 𝑘𝑓_0𝑛1+𝛽𝑘², where 𝛽 is the inharmonicity coefficient. Typical values for coefficient 𝛽 range from 10⁻⁴ to 10⁻³ in piano bass notes [10]. Partial frequencies exceed the selected frequency range from 𝑘=25 for 𝛽=10⁻⁴, 𝑘=11 for 𝛽=5∗10⁻⁴, or 𝑘=8 for 𝛽=10⁻³. The fundamental frequency 𝑓0_𝑛 of the MIDI note n is calculated as 𝑓_0𝑛=440⋅2^{(𝑛−69)/12}. (3.3) The analysis above indicates a problem that harmonics of one note may not just be at the integer multiple position of the fundamental frequency because of the inharmonicity. According to this, when we calculate 𝐻_𝑛, we use 𝑎^′_𝑘 instead of 𝑎_𝑘, where 𝑎^′_𝑘 is the maximum partial amplitude found in the frequency range [𝑘𝑓₀⋅2^−(1/24),𝑘𝑓₀⋅2^1/24]. When there are no partials in the frequency range, 𝑎^′_𝑘 is set to 0.

In the identification stage, the harmonic structure matching rate can be used to determine the candidate fundamental frequency.

When playing a note, the spectrum will contain its entire harmonics. If the 𝑓0 is located at the same semitone as the note 𝑛, it will have the same harmonic structure as 𝐻_𝑛, in which the harmonic matching ratio of the kth harmonic can be defined as the following: 𝑟𝑓₀=𝑌𝑓,𝑘^𝑘𝑎_𝑘, (3.4) where 𝑌(𝑓_𝑘) is the STFT of music and 𝑎_𝑘 is the 𝑘-th element in 𝐻_𝑛.

In order to remove the influence of harmonic overlap in the polyphony music, the minimal harmonic matching ratio of all harmonic components is chosen as the harmonic matching ratio of this note, which just as shown in 𝑟𝑓₀=min𝑟𝑓₀,𝑘|^𝑛_𝑘=1. (3.5) And, then, in order to better describe the situation of playing the notes, the strength of notes can be defined as follows: 𝑆𝑓₀=^𝑛_𝑘=1𝑟𝑓₀𝑎_𝑘. (3.6) The larger 𝑆 shows the larger probability of including the note which has a fundamental frequency of 𝑓₀. If 𝑆 is larger than the threshold th2, the note whose fundamental frequency is 𝑓₀ will be chosen as a candidate fundamental frequency. When all the candidates are chosen, the frequency primary selection step is completed.

4. Other Mathematical Treatments in This Algorithm

4.1. Principal Component Analysis

In training stage and identification stage, PCA is an important step to reduce dimensions, so we introduce the calculate process in detail here.

Principal component analysis (PCA) is a mathematical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables. This transformation is defined in such a way that the first principal component has as high a variance as possible (i.e., accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it be orthogonal to (uncorrelated with) the preceding components. Principal components are guaranteed to be independent only if the data set is jointly normally distributed. PCA is sensitive to the relative scaling of the original variables.

PCA is mathematically defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by any projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on. PCA is a mature tool which has classic calculation process. The implementation method of PCA can be found in [10].

The calculation steps of PCA are as follows.(1) Calculate the correlation coefficient matrix: 𝑟𝑅=¹¹𝑟₁₂⋯𝑟_1𝑝𝑟₂₁𝑟₂₂⋯𝑟_2𝑝𝑟⋮⋮⋮⋮_𝑝1𝑟_𝑝2⋯𝑟_𝑝𝑝, (4.1) where 𝑟_𝑖𝑗(𝑖,𝑗=1,2,…,𝑝) is correlation coefficient of original variables 𝑥_𝑖 and 𝑥_𝑗. The calculation formula is 𝑟_𝑖𝑗=∑^𝑛_𝑘=1𝑥_𝑘𝑖−𝑥_𝑖𝑥_𝑘𝑗−𝑥_𝑗∑^𝑛_𝑘=1𝑥_𝑘𝑖−𝑥_𝑖²∑^𝑛_𝑘=1𝑥_𝑘𝑗−𝑥_𝑗². (4.2)𝑅 is a real symmetric matrix, that is, 𝑟_𝑖𝑗=𝑟_𝑗𝑖, so we only need to calculate the upper triangular elements or lower triangular elements.(2) Calculate the eigenvalues and eigenvectors: first, solve the characteristic equation |𝜆𝐼−𝑅|=0. Usually the Jacobi method is used to find out the eigenvalues 𝜆_𝑖(𝑖=1,2,…,𝑝), and let them be arranged in order of size, that is, 𝜆₁≥𝜆₂≥⋯≥𝜆_𝑝≥0, and find out the eigenvectors 𝑒_𝑖(𝑖=1,2,…,𝑝) corresponding to the eigenvalues 𝜆_𝑖, respectively. Here ‖𝑒_𝑖‖=1, that is, ∑^𝑝_𝑗=1𝑒²_𝑖𝑗=1, where 𝑒_𝑖𝑗 represent the 𝑗 component of vector 𝑒_𝑖.(3) Calculate the contribution of the main components and cumulative contribution. The contribution of main components 𝑧_𝑖 is 𝜆_𝑖∑^𝑝_𝑘=1𝜆_𝑘𝑖=1,2,…,𝑝. (4.3) The cumulative contribution is ∑^𝑖_𝑘=1𝜆_𝑘∑^𝑝_𝑘=1𝜆_𝑘𝑖=1,2,…,𝑝. (4.4)(4) Calculate the weight of the main components: 𝑙_𝑖𝑗=𝑝𝑧_𝑖,𝑥_𝑗=𝜆_𝑖𝑒_𝑖𝑗𝑖,𝑗=1,2,…,𝑝. (4.5) After it, we can calculate the score of each main components: 𝑧𝑍=¹¹𝑧₁₂⋯𝑧_1𝑚𝑧₂₁𝑧₂₂⋯𝑧_2𝑚𝑧⋮⋮⋮⋮_𝑛1𝑧_𝑛2⋯𝑧_𝑛𝑚, (4.6) where 𝑍_𝑚=𝑙_𝑚1𝑥₁+𝑙_𝑚2𝑥₂+⋯+𝑙_𝑚𝑝𝑥_𝑝. (4.7)

4.2. Mathematical Representation in Multiple-F0 Estimation

Multiple fundamental frequency (multiple-F0) estimation is used to estimate multiple notes which are sounded at the same time in music. It is the core technology and the main difficulties in automatic music transcription system. Most actual music are polyphony ones, which means there are more than one notes at the same time. The situation is much more complex when the spectrums of the notes are added and it makes the multiple-F0 estimation more difficult. To improve the performance of the system, a good multiple-F0 estimation algorithm needs to consider many factors, such as inharmonicity factor, frequency missing, harmonic missing, frequency overlap, and frequency error [6]. The algorithm proposed in this paper establishes a mathematical model for multiple fundamental frequency estimation. To obtain the result of multiple-F0 estimation is equivalent to solve this math problem. In this part, we introduce the main idea of this algorithm.

In the music division rules, there is sufficient frequency separation between notes, and the masking effect is relatively weak among pure tones. Thus, the masking effect can be ignored while the requirement of computational accuracy is not very highly. Ignoring the masking effect, the loudness can be added with a linear characteristic. It means that the loudness of the sound mixing multiple notes is the sum of the loudness of each note.

From the above analysis we can see that the audio to be estimated is the linear combination of standard notes. Suppose that the number of notes is 𝑛, each note includes 𝑚 features after PCA process, and then the training set 𝑆 is an 𝑛×𝑚 matrix. The audio to be estimated after preprocessing, PCA, and normalization is recorded as 𝑌, then 𝑌 is a 1×𝑚 vector, and 𝑚 is the characteristic dimension. If the energy lost in PCA is recorded as Sr, we get the following: 𝑋⋅𝑆+𝑆𝑟=𝑌, (4.8) where 𝑆 is the result set of the training process and it is an 𝑛×𝑚 matrix. X is a 1×𝑛 vector, 𝑋=[𝑥₁,𝑥₁,…,𝑥_𝑛], and it presents the combination coefficient corresponding to each note. If we regard the Sr as an error part, we can ignore it and get the next formula: 𝑋⋅𝑆≈𝑌. (4.9)

The main task of multiple-F0 estimation is to estimate the best 𝑋 to make 𝑋𝑆 as more close as 𝑌. This is a problem of computing extreme and it can be resolved by the knowledge of linear algebra.

Let 𝑓(𝑥)𝑥=𝑋𝑆−𝑌=¹𝑠₁₁+𝑥₂𝑠₂₁+⋯+𝑥_𝑛𝑠_𝑛1−𝑦₁⋮𝑥₁𝑠_1𝑚+𝑥₂𝑠_2𝑚+⋯+𝑥_𝑛𝑠_𝑛𝑚−𝑦_𝑚=𝑓₁𝑓₂⋮𝑓_𝑚,𝑔(𝑥)=𝑓^𝑇(𝑥)𝑓(𝑥)=𝑓²₁+𝑓²₂+⋯+𝑓²_𝑚. (4.10)

The problem is to obtain a best 𝑋 to make 𝑓(𝑥) get the maximum value. It can be proved that 𝑔(𝑥) and 𝑓(𝑥) get the maximum value at the same place. So the next problem is to calculate 𝑋 to make 𝑔(𝑥) gets the maximum value. According to the knowledge of higher mathematics we can get 𝜕𝑔(𝑥)𝜕𝑥₁=2𝑓₁𝑠₁₁+2𝑓₂𝑠₁₂+⋯+2𝑓_𝑚𝑠_1𝑚(𝑥)=0𝜕𝑔𝜕𝑥₂=2𝑓₁𝑠₂₁+2𝑓₂𝑠₂₂+⋯+2𝑓_𝑚𝑠_2𝑚⋮(𝑥)=0𝜕𝑔𝜕𝑥_𝑛=2𝑓₁𝑠_𝑛1+2𝑓₂𝑠_𝑛2+⋯+2𝑓_𝑚𝑠_𝑛𝑚=0. (4.11)

Equations (4.12) can be obtained after the decomposition (4.11): 𝑥₁,𝑥₂,…,𝑥_𝑛⋅𝑆₁𝑆₂⋯𝑆_𝑛^𝑇⋅𝑆^𝑇₁−𝑌⋅𝑆^𝑇₁𝑥=0₁,𝑥₂,…,𝑥_𝑛⋅𝑆₁𝑆₂⋯𝑆_𝑛^𝑇⋅𝑆^𝑇₂−𝑌⋅𝑆^𝑇₂⋮𝑥=0₁,𝑥₂,…,𝑥_𝑛⋅𝑆₁𝑆₂⋯𝑆_𝑛^𝑇⋅𝑆^𝑇_𝑛−𝑌⋅𝑆^𝑇_𝑛=0. (4.12) Simplification of (4.12) can get 𝑋⋅𝑆⋅𝑆^𝑇₁−𝑌⋅𝑆^𝑇₁=0𝑋⋅𝑆⋅𝑆^𝑇₂−𝑌⋅𝑆^𝑇₂⋮=0𝑋⋅𝑆⋅𝑆^𝑇_𝑛−𝑌⋅𝑆^𝑇_𝑛=0. (4.13)Write (4.13) into the matrix form: 𝑋⋅𝑆⋅𝑆^𝑇=𝑌⋅𝑆^𝑇, (4.14) where 𝑆 and 𝑌 are known, so the 𝑋 can be obtained by knowledge of linear algebra.

It should be noted that if there are some negative coefficients in X, it means that the audio will not contain the corresponding note and we should calculate this again without this note. It is repeated until all components of X are positive or repeated 𝑡 times (to avoid an infinite loop, you need to manually set up a number of cycles t). A threshold 𝑥₀ can be set. If 𝑥_𝑖>𝑥₀, it indicats that the note 𝑖 is included in this frame. In this way, we can estimate all the notes contained in this frame.

We can solve the function (4.9) in another way. In function (4.9), 𝑆 is an 𝑛×𝑚 matrix. In general case, 𝑚>>𝑛, so function (4.9) is an ill-posed matrix equation. A truncation completely least squares method can be used to get the solution of this function because this method is not sensitive to error. The step-by-step process is given as follows.

At first, the singular value decomposition of augmented matrix [𝑆,𝑌] needs to calculated: [𝑆,𝑌]=𝑃^𝜆𝑄^𝑇=^𝑛+1_𝑖=1𝑝_𝑖𝜆_𝑖𝑞_𝑖,𝜆₁>𝜆₂>⋯>𝜆_𝑛+1. (4.15) And then, select a truncation strategy 𝑘≤min{𝑛,rank(𝑆,𝑌)} and cut off the smaller singular value and let 𝑄^′=𝑞_{𝑛+1,𝑘+1},𝑞_{𝑛+1,𝑘+2},…,𝑞_{𝑛+1,𝑛+1}≠0. (4.16) Third, let 𝑛^′=𝑛−𝑘+1, and make Q become a partitioned matrix: 𝑄𝑄=¹¹𝑄₁₂𝑄₂₁𝑄₂₂, (4.17) where 𝑄₁₁∈𝐶^𝑛×𝑘,𝑄₁₂∈𝐶^{𝑛×𝑛′},𝑄₂₁∈𝐶^1×𝑘,𝑄₂₂∈𝐶^1×𝑛′.

At last we can get the solution: 𝑋=−𝑄₁₂𝑄^∗₂₂=−𝑄₁₂𝑄^𝑇₂₂𝑄₂₂₂⁻², (4.18) where 𝑄^∗₂₂=𝑄^𝑇₂₂‖𝑄₂₂‖₂⁻² is a generalized inverse matrix.

After we get X, the negative and smaller value components were deleted and the rest ones are the mixing coefficients of the corresponding notes.

We can use both the ways to solve function (4.9) and make them mutual authentication to make an improvement.

When calculating 𝐻_𝑛 in training stage we consider with the information of frequency missing, harmonic missing, and while in frequency primary selection stage we consider with inharmonicity factor and frequency error. Function (4.9) can indicate the information of frequency overlap. As a result, our algorithm has a better result than others.

4.3. Filer Based on the Music Category and Priori Probability

This content is one of the most contribution of this paper.

Just as we know, there are kinds of music actually, such as new age, rhythm and blues, rap, rock, and jazz. Different music has different rhythm and melody and many other things. From the audio features, they have different low short time energy ratio and nonzero pitch ratio and other characteristics. The probability of occurrence of each note is different for different types of music. The probability distribution of the same note in different types of music is also different. We can use this to perform a filer based on the music category for the previous experimental results.

It is needed to point out that to classify the music is not the research contents of this paper. We assume that all the category of each music belongs to are known.

We only consider the most popular notes (about 72) in actual songs. Let event 𝐴_𝑖𝑥 = {note 𝑥 is included in the ith frame}, 𝐴_(𝑖+1)𝑦 = {note 𝑦 is included in the(𝑖+1) th frame}. It is obvious that the number of x and y is limited, and the number of event 𝐴_𝑖𝑥 or 𝐴_(𝑖+1)𝑦 is also limited. Let event 𝐶_𝑥𝑦 = {while current frame contains note 𝑥, the next frame contains note 𝑦}. According to the definition of conditional probability we can get that 𝑃(𝐶_𝑥𝑦)=𝑃(A_(𝑖+1)𝑦∣𝐴_𝑖𝑥). For different kind of music, the conditional probability 𝑃(A_(𝑖+1)𝑦∣𝐴_𝑖𝑥) is not the same for the same note x or y. If we know the kind of the music, we can filter the following frames by the corresponding conditional probability. In this way, we can remove the notes with small probability and validate the notes in current frame through the relevant posterior probability.

In the training process, we can get the 𝑃(𝐴_𝑖𝑥) through experiment by 𝑃(𝐴_𝑖𝑥) = {the number of frames including note 𝑥}/{the number of total frames}. We can also get 𝑃(𝐶_𝑥𝑦)=P(𝐴_(𝑖+1)𝑦∣𝐴_𝑖𝑥) = {the number of the next frame contains note 𝑦 while current frame contains note 𝑥}/{the number of total frames}.

In the identification process we can calculate the following data.(i)Get the 𝑃(𝐴_(𝑖+1)𝑦) by querying training result set directly. (ii)Get the 𝑃(𝐴_(𝑖+1)𝑦∣𝐴_𝑖𝑥) by querying training result set directly. (iii)Calculate the 𝑃(𝐴_(𝑖+1)𝑦) by the whole probability formula: 𝑃𝐴_(𝑖+1)𝑦=⁷²_𝑥=1𝑃𝐴_𝑖𝑥𝑃𝐴_(𝑖+1)𝑦∣𝐴_𝑖𝑥. (4.19)(iv)Calculate the 𝑃(𝐴_𝑖𝑥∣𝐴_(𝑖+1)𝑦) by the Bayesian formula: 𝑃𝐴_𝑖𝑥∣𝐴_(𝑖+1)𝑦=𝑃𝐴_𝑖𝑥𝑃𝐴_(𝑖+1)𝑦∣𝐴_𝑖𝑥∑⁷²_𝑥=1𝑃𝐴_𝑖𝑥𝑃𝐴_(𝑖+1)𝑦∣𝐴_𝑖𝑥. (4.20)

Analysis of the above four probabilities can be seen. The first probability is a priori probability without any restrictions. The second one has been considered with the relation of the adjacent frames. The third one is calculated with the whole probability formula and the forth is a posterior probability calculated by the Bayesian formula. The first three probabilities are priori probabilities used to directly determine the probability of the current frame that contained a note. The fourth one is a posterior probability used to determine the probability of the previous frame that contained one note while the current frame contained another one. Each probability can be used to get the result alone but we put the four probabilities together to make the result more accurate.

Setting a probability threshold 𝑃, we consider the event 𝐴_(𝑖+1)𝑦 as true only when all of the probabilities mentioned above are larger than 𝑃.

5. Experimental Methods and Results

According to the algorithm described in the previous section, we did simulation in Matlab. The experiment parameters in each step are listed as follows.(i)All audio signals are decomposed at a sampling frequency of 𝑓𝑠=22.5 kHz and 16 bit. (ii)Hamming windowing. (iii)FFT length 𝐿=1024.(iv)FIR High Pass Filter 𝐻(𝑧)=1−𝛼𝑧⁻¹,𝛼=0.98, used to pre-emphasis. (v)Frame length is 100 ms. (vi)72 notes played by piano are used to train, which from C3 (note 36) to B8 (note 107), and each note we get 100 wave-form data to train. (vii)The number of songs used to training is 100 for each category, and do statistics and get 𝑃(𝐴_(𝑖+1)𝑦) and 𝑃(𝐴_(𝑖+1)𝑦∣𝐴_𝑖𝑥) base on them. (viii)The PCA score 𝐼_pca = 80%. (ix)The threshold of spectral envelope in frequency primary selection step th1 = 0.1. (x)The threshold of intensity coefficient: th2 = 1.5. (xi)The probability threshold of filter step is 𝑃 = 0.6. (xii)We obtain note error rate (NRE) as follows: 𝐸=FP+FN,Notes (5.1)where FP (false positives) is the number of inactive note frames transcribed as active and FN (false negatives) is the number of active note frames transcribed as inactive.

In order to fully test the performance of the algorithm presented in this paper, we made a lot of experiments with different types and styles of music, including classic music, jazz, nursery rhyme, soothing music, fast-paced music, and the music which has a lot of fundamental frequency overlapping and the music which has just a little fundamental frequency overlapping. The size of a frame has a little influence on the recognition results of different types of music. Analysing of the results made us found that our algorithm was not sensitive about changes in the rhythm of music. Both fast rhythm music and slow rhythm have similar identification results, but it is sensitive with the change rules of rhythm. The music with more regular rhythm changes can get better recognition results.

A number of experiments show that our algorithm has an average NER about 13%. That is a very high accuracy rate compared with some state-of-the-art algorithms described in [11, 12]. The transcription result has no obvious impact on understanding this music for us.

6. Conclusion and Perspective

This paper presents a compositional pattern recognition and machine learning methods for computer-synthesized music specifically to multiple-F0 estimation and builds an efficient automatic music transcription system. This method also considers the human auditory model and the harmonic structure of music notes and improves the algorithm based on this. Although using the harmonic match and iterative delete alone can finish the multiple-F0 estimation, or deleting the frequency primary selection step in identification stage and establishing the mathematical model and solving it can also finish the task, combining the two methods can improve the performance of the algorithm and the recognition result. The usage of Bayesian estimation and a priori probability improved the performance a lot. Furthermore, for all this, something which needs to be improved must be pointed out. First, we can improve the definition of harmonic matching to get a more accurate result. And then, for FFT, we can consider using changeable FFT length, because the notes in the low-frequency region are rich in harmonic components while the notes in high-frequency region have relatively simple harmonic components. In low-frequency region, a higher frequency resolution is needed while in high-frequency region the situation is just the opposite. Third, the truncation threshold th1 or th2 in our algorithm can be defined in a changeable way to get a more accurate result. In addition, because ensemble music includes many instruments, we need to train with many kinds of notes and it has a large amount of computation. If we can develop an unsupervised method and the prior information about the instrument is not obtained from files of the same instrument in different music databases but is directly obtained from the music file to be analyzed, that will be improve efficiency a lot.

Acknowledgments

Thanks are due to Professor Hou Xiaorong for meticulous guidance all the time and to Professor Tang Jiyong for financial support of the authors’ research.