Time Domain Method for Precise Estimation of Sinusoidal Model Parameters of Co-Channel Speech

Abstract

A time domain method to precisely estimate the sinusoidal model parameters of cochannel speech is presented. The method does not require the calculation of the Fourier transform nor the multiplication by a window function. It incorporates a least-squares estimator and an iterative technique to model and separate the cochannel speech into its individual speakers. The application of this method on speech data demonstrates the effectiveness of this method in separating cochannel speech signals in different target-to-interference ratios. This method is capable of producing accurate and robust parameter estimation in low signal-to-noise ratio situations compared to other existing algorithms.

1. Introduction

Separation of mixed speech signals is still one of the major challenges in speech processing. This problem is commonly referred to as co-channel speech separation. The main goal of co-channel speech separation is to automatically process the mixed signal in order to recover each talker's original speech. Minimizing artifacts in the processed speech is a key concern, especially if the final goal is to use the recovered speech in machine-based applications such as automatic speech recognition and speaker identification systems.

Several previous studies have developed signal processing algorithms for modeling and separating co-channel speech. The primary approaches have taken the harmonic structure of voiced speech as the basis for separation and have used either frequency-domain spectral analysis and reconstruction [1–3] or time-domain filtering [4]. One promising approach to address co-channel speech separation is to exploit a speech analysis/synthesis system based on sinusoidal modeling of speech. For example, in [1, 2] a voiced segment of co-channel speech is modeled as the sum of harmonically related sine waves with constant amplitudes, frequencies, and phases. In the sinusoidal modeling approach, the speech parameters of individual talkers are estimated by applying a high-resolution short-time Fourier transform (STFT) to the windowed speech waveform. The frequencies of underlying sine waves are assumed to be known a priori from the individual speech waveforms, or they are determined by using a simple frequency domain peak-picking algorithm. The amplitudes and phases of the component waves are then estimated at these frequencies by performing a least-squares (LS) algorithm. This technique has the following drawbacks:(1)the accuracy of the estimate is limited by the frequency resolution of the STFT;(2)error is introduced due to edge effects of the window function used for the STFT. This paper presents a time domain method to precisely estimate the sinusoidal model parameters of co-channel speech. The method does not require the calculation of the STFT nor the multiplication by a window function. It incorporates a time-domain least-squares estimator and an adaptive technique to model and separate the co-channel speech into its individual speakers. The performance of the proposed method is evaluated using a database consisting of a wide variety of mixed male and female speech signals at different target-to-interference ratios (TIRs).

This paper is organized as follows. In Section 2, the sinusoidal model of co-channel speech consisting of 𝐾 speakers is presented. The proposed time-domain method for estimating the sinusoidal model parameters is discussed in Section 3. In Section 4, experimental results and comparisons with other techniques are reported and discussed. Finally, the results are summarized and conclusions given in Section 5.

2. Sinusoidal Modeling of Co-Channel Speech

According to the speech analysis/synthesis approach based on the sinusoidal model [1], a short segment of co-channel speech (about 20 to 30 milliseconds) can be represented as the sum of harmonically related sinusoidal waves with constant amplitudes, frequencies, and phases as follows: 𝑥(𝑛)=^𝐾_𝐿𝑘=1^𝑘_ℓ=1=𝑐^ℓ(𝑘)cos(ℓ𝜔_𝑘𝑛−𝜙^ℓ(𝑘))(1)_𝐾_𝐿𝑘=1^𝑘_ℓ=1[𝑎^ℓ(𝑘)cosℓ𝜔_𝑘𝑛+𝑏^ℓ(𝑘)sinℓ𝜔_𝑘𝑛],(2) where 𝑛=0,…,𝑁−1 is the discrete time index, 𝜔_𝑘 is the fundamental frequency for that segment of the 𝑘th talker, and 𝑐^ℓ(𝑘), ℓ𝜔_𝑘, and 𝜙^ℓ(𝑘) denote the amplitude, frequency, and phase, respectively, of the ℓth harmonic of the 𝑘th talker. The total number of harmonics in each talker's model is given as 𝐿_𝑘 for 𝑘=1,…,𝐾. The quadrature amplitudes 𝑎^ℓ(𝑘) and 𝑏^ℓ(𝑘) in (2) are related to 𝑐^ℓ(𝑘) and 𝜙^ℓ(𝑘) in (1) as follows [1]:𝑐^ℓ(𝑘)=(𝑎^ℓ(𝑘))²+(𝑏^ℓ(𝑘))²,𝜙^ℓ(𝑘)=tan⁻¹(𝑏^ℓ(𝑘)𝑎^ℓ(𝑘)).(3) A precise estimate of the sinusoidal-model parameters is essential for separating the co-channel speech into its individual components. The basic problem addressed in this paper can be stated as follows. Given the real observed 𝑁 samples of the co-channel speech sequence 𝑥(𝑛), find the parameters 𝐿_𝑘, 𝜔_𝑘, {𝑎̂^ℓ(𝑘)}𝐿^𝑘_ℓ=1, and {𝑏̂^ℓ(𝑘)}𝐿^𝑘_ℓ=1 that form the sequence, 𝑥̂𝑛=^𝐾_𝑘=1𝐿_𝑘_ℓ=1[𝑎̂^ℓ(𝑘)cosℓ𝜔_𝑘𝑛+𝑏̂^ℓ(𝑘)sinℓ𝜔_𝑘𝑛],(4)that best fits 𝑥(𝑛) by minimizing the mean-squared error (MSE)𝐸=1𝑁^𝑁−1_𝑛=0𝑥𝑛−𝑥̂𝑛².(5)In the following sections, we will consider the case of two talkers (𝐾=2) to represent the co-channel speech without loss of generality.

3. Time-Domain Estimation of Model Parameters

3.1. Estimation Setup

In a matrix notation, we may write (4) as𝐱̂=𝐐𝐡,(6)where 𝐱̂ is the vector 𝐱̂=𝑥̂(0),𝑥̂(1),…,𝑥̂𝑁−1^𝑇,(7) and 𝐡 is given as 𝐡=𝐡⁽¹⁾𝐡⁽²⁾,(8)with𝐡^(𝑘)=[𝑎̂^1(𝑘),𝑎̂^2(𝑘),…,𝑎̂^(𝑘)𝐿_𝑘,𝑏̂^1(𝑘),𝑏̂^2(𝑘),…,𝑏̂^(𝑘)𝐿_𝑘]^𝑇.(9)𝐐 is a matrix of the form 𝐐=𝐐⁽¹⁾𝐐⁽²⁾,(10)where the matrix elements are given as 𝑄^{(𝑘)𝑖𝑗}=⎧⎪⎨⎪⎩cos𝑖𝑗𝜔^𝑘sin𝑖𝑗−𝐿_𝑘for𝑗=1,2,…,𝐿^𝑘,𝜔^𝑘for𝑗=𝐿^𝑘+1,…,2𝐿_𝑘,(11)with 𝑖=0,1,…,𝑁−1 and 𝑘=1,2. The MSE in (5) can now be written as 𝐸=‖‖𝐱−𝐱̂‖‖²²=𝐱^𝑇𝐱+𝐱̂^𝑇𝐱̂−2𝐱̂^𝑇𝐱,(12) where 𝐱=𝑥(0),𝑥(1),…,𝑥(𝑁−1)^𝑇.(13)Substituting (6) into (12) gives𝐸=𝐱^𝑇𝐱+𝐡^𝑇𝐐^𝑇𝐐𝐡−2𝐡^𝑇𝐐^𝑇𝐱.(14) The estimation criteria are to seek the minimization of (14) over the parameters 𝐿_𝑘, 𝜔_𝑘, {𝑎̂^ℓ(𝑘)}𝐿^𝑘_ℓ=1, and {𝑏̂^ℓ(𝑘)}𝐿^𝑘_ℓ=1.

The most important and difficult part in the estimation process is to estimate the fundamental frequencies {𝜔_𝑘}_𝑘=1,2. Unfortunately, without a priori knowledge of the frequency parameters, direct minimization of (14) is a highly nonlinear problem that is very difficult to solve. If these frequencies were known a priori or can be estimated precisely, one can easily find the optimum values of the other parameters accordingly.

3.2. Estimating the Number of Harmonics

If the fundamental frequencies 𝜔_𝑘 are assumed to be known, the total number of harmonics in each signal can be estimated simply as𝐿_𝑘=𝜋𝜔_𝑘.(15)Practically, 𝐿_𝑘 is chosen much smaller than the value calculated by (15) since most of the energy of voiced speech is concentrated below 2 kHz. Using this assumption can reduce dramatically the computational complexity of the system.

3.3. Estimating the Amplitude Parameters

The optimum values of the quadrature parameters {𝑎^ℓ(𝑘)}^𝐿𝑘_ℓ=1 and {𝑏^ℓ(𝑘)}^𝐿𝑘_ℓ=1 can be estimated directly (assuming the availability of the fundamental frequencies) by finding the standard linear LS solution to (14) as follows [5]:𝐡_opt=𝐐^𝑇𝐐⁻¹𝐐^𝑇𝐱=𝐑⁻¹𝐏,(16)where𝐑=𝐐^𝑇𝐐,𝐏=𝐐^𝑇𝐱.(17) The minimum MSE corresponding to 𝐡_opt is given by substituting (16) into (14) to give 𝐸_min=𝐱^𝑇𝐱−𝐏^𝑇𝐑⁻¹𝐏.(18)

3.4. Estimating the Fundamental Frequencies

Since in practical applications, the fundamental frequencies of the individual speech waveforms are not known a priori, they must be estimated from the mixed data. A direct approach to solve this problem is to search the 𝐾-dimensional MSE surface for its minimum with respect to the fundamental frequencies. The initial estimate can be determined either from the previous frame or by applying a simple rough multipitch estimation method such as the one proposed in [6], which is a time-domain method that depends on the average magnitude difference function. After finding an initial guess for the 𝜔_𝑘, the optimum fundamental frequencies can be estimated by searching the MSE surface of (18) by the method of steepest descent [7]. Using a weight vector 𝐰=[𝜔₁,𝜔₂]^𝑇, we describe the steepest descent algorithm by 𝐰(𝑖+1)=𝐰(𝑖)−12𝜇∇𝐄(𝑖),(19) where −∇𝐄(𝑖)=−∇𝐄⁽¹⁾−∇𝐄₍₂₎(𝑖)(𝑖),(20)and 𝜇 is a positive scalar that controls both the stability and the speed of convergence. The gradient of the MSE is calculated by differentiating (18) with respect to each fundamental frequency as follows:−∇𝐄^(𝑘)=−𝜕𝐸^min𝜕𝜔_𝑘=𝐱^𝑇𝐐̇𝐡_opt−𝐡^𝑇_opt𝐑̇𝐡_opt+𝐡^𝑇_opt𝐐̇^𝑇𝐱,(21)where𝐐̇=𝜕𝐐𝜕𝜔_𝑘,𝐑̇=𝜕𝐑𝜕𝜔_𝑘.(22) Differentiating (10) and substituting into (21) give −∇𝐄^(𝑘)=2𝐱−𝐐𝐡_opt^𝑇𝐐̇^(𝑘)𝐡^(𝑘)_opt.(23) The fundamental frequencies are updated iteratively using (19). After each iteration, the optimum amplitude parameters corresponding to the estimated frequencies are calculated using (16). Note that even by using (19), final estimates of fundamental frequencies may still have small inaccuracies because frequencies may vary slightly within the speech frame. The use of exact gradient to update the fundamental frequencies in (19) gives an advantage compared to [1], where an approximation of the gradient is used. Gradient calculation is an integrated process in the time-domain method since the components on the right-hand side of (23) are already part of the previous steps in the algorithm.

An example of the MSE surface obtained for the single talker (𝐾=1) case is shown in Figure 1. Figure 1(a) shows a 30-millisecond speech frame for a single talker, while Figure 1(b) shows the corresponding MSE surface using (18) as the cost function. From Figure 1(b), the optimal fundamental frequency is approximately 165 Hz. For the two-talker case (𝐾=2), the MSE surface would instead be two-dimensional.

(a)

(b)

Figure 1 

An example of MSE surface for a single-talker: (a) 30-millisecond single voiced speech segment in the time domain, (b) MSE performance versus fundamental frequency based on (18).

3.5. The Ill-Conditioned Estimation Problem

In some instances, the harmonics of the two speakers can be very close to each other. When the harmonics overlap, the matrix 𝐑 in (17) will be singular, and the parameter estimation process in (16) becomes ill-conditioned. To handle this problem, the spacing between adjacent harmonics is continuously calculated. If two adjacent harmonics are found to be closely spaced, that is, less than 25 Hz apart, only one sinusoid is used to represent these two harmonics. The amplitude parameters of this single component are then estimated and shared equally between the two speakers [1].

4. Simulation Results

The performance of the proposed method is evaluated using a speech database consisting of 200 frames of mixed speech. All-voiced speech segments of length 30 milliseconds were randomly chosen from the TIMIT dataset [8] for male and female speakers and mixed at different TIRs. The speech data were sampled at a rate of 16 kHz.

Two sets of simulations were conducted to compare the performance of the proposed method with the frequency sampling approach presented in [1]. As suggested by the authors, a Hann window and a high-resolution STFT of length 𝑀=4096 were used in the frequency-domain technique. To avoid errors due to the multipitch detection algorithm, the initial guess of the fundamental frequency of each talker was calculated directly from the original speech frames before mixing, using a simple autocorrelation method.

In the first set of simulations, the comparison was carried out in terms of the signal-to-distortion ratio (SDR) versus TIR as shown in Figure 2 for TIRs ranging from −5 to +15 dB. The SDR measure is defined as [9] SDR_dB=10log₁₀∑_𝑛𝑠(𝑛)²∑_𝑛𝑠(𝑛)−𝑠(̂𝑛)²,(24) where 𝑠(𝑛) is the original target signal before mixing, and 𝑠(̂𝑛) is the reconstructed signal after separation from the mixture 𝑥(𝑛). Each point in the plot of Figure 2 presents the ensemble average of the SDRs over all 200 test frames. Two cases are considered for each algorithm. In case 1, precise estimation of the fundamental frequencies is done using (19), and in case 2 only the initial guess of the fundamental frequencies is used. Plots SDR-TD1 and SDR-TD2 are the results for the proposed time-domain algorithm in case 1 and case 2, respectively, while the plots SDR-FD1 and SDR-FD2 depict the results for the frequency-domain method. As can be seen from Figure 2, the SDR increases monotonically for both algorithms with an increase of the TIR in all cases.

Figure 2 

SDR results; SDR-TD1 and SDR-TD2 for the proposed time-domain method, and SDR-FD1 and SDR-FD2 for the frequency-domain method [1], with precise and initial frequency estimates of {𝜔𝑘}𝑘=1,2, respectively.

More importantly, we see from Figure 2 that the proposed technique outperforms the frequency-domain techniques in both case 1 and case 2. At TIR = −5 dB, SDR-TD1 and SDR-TD2 are greater than SDR-FD1 and SDR-FD2 by about 2 and 1 dB, respectively. This difference is greater for larger TIR. As suggested in Section 1, analysis of the resulting estimates using voiced speech segments has revealed that the discrepancies are due to the limited frequency resolution of the STFT (even with 𝑀=4096) and due to the choice of window function and resulting edge effects. Other window functions such as rectangular and Hamming windows had similar discrepancies when tested.

The robustness against background noise was examined in a second set of simulations using MSE versus signal-to-noise ratio (SNR). Speech segments were corrupted by additive white Gaussian noise (AWGN) with SNR varied from 0 to 15 dB. The results are presented in Figure 3. As shown in the figure, the proposed algorithm has a superior performance in low SNR compared to the frequency-domain technique. The AWGN causes additional frequency resolution problems after even a high-resolution STFT. If the proposed time-domain estimation approach is used instead, then the effect of the AWGN is not as severe.

Figure 3 

MSE results for AWGN for both the proposed time-domain technique compared with the standard frequency-domain method [1].

5. Conclusions

A time-domain method to precisely estimate the sinusoidal model parameters of co-channel speech is presented. The method does not require calculation of the STFT nor multiplication by a window for the primary model parameters. The proposed method incorporates a least-squares estimator and an adaptive technique to model and separate the co-channel speech into its individual speakers, all in the time domain.

The application of this time-domain method on real data demonstrates the effectiveness of this method in separating co-channel speech signals at different TIRs. Overall, an improvement of 1–3 dB in SDR is obtained over the frequency-domain method, dependent on the accuracy of the fundamental frequency estimates of the talkers in the tested two-talker scenario. Note that these time-domain results are compared with the frequency-domain approach with an 𝑀=4096-point STFT. Changes in 𝑀 would affect the precision in the estimates of the frequency-domain technique.

We also note that the time-domain method is not as sensitive to additive white Gaussian noise as is the frequency-domain method for sinusoidal modeling. This result is particularly true for lower-SNR situations.

Acknowledgment

The authors wish to thank the National Capital Institute of Telecommunications (NCIT) for partially funding this research.