Abstract

Fault diagnosis is an important means to ensure the safe and reliable operation of mechanical equipment. In machinery fault diagnosis, collecting and mining the potential fault information of the vibration signal is the most commonly used method to reflect the operating status of the equipment. In engineering scenarios, in the face of rotating machinery with variable speed, simple time domain analysis or frequency domain analysis is difficult to solve the problem. The time-frequency analysis technology that combines time-frequency transformation and data analysis can solve practical engineering problems by capturing the transient information of the signal. At present, a large number of related literatures have been published in academic journals. This paper hopes to provide convenience for relevant researchers and motivate researchers to further explore by summarizing the published literature. First, this paper briefly explains the concept of time-frequency analysis and its development. Then, the time-frequency transformation method proposed for the characteristics of rotating machinery fault vibration signal and related works of literature are reviewed, and the key issues of the application of time-frequency transformation method in rotating machinery fault diagnosis are discussed. Next, this paper summarizes the relevant literature on the combination of data analysis technology and time-frequency transformation and sorts out its development route and prospects. The study reveals that time-frequency analysis technology is able to detect the rotating machinery fault effectively. The time-frequency analysis technology has made abundant achievements in the field of rotating machinery fault diagnosis. It is expected that this review would inspire researchers to explore the potential of time-frequency analysis as well as to develop advanced research in this field.

1. Introduction

In fault diagnosis of rotating machinery, infrared image [13] and acoustic feedback can [4, 5] be used as important indicators to judge the health status of the machine. But due to the high cost of the infrared imaging and acoustic emission, the analysis of mechanical vibration signals is the most commonly used method in mechanical fault diagnosis [6]. Mechanical systems often have complex structures. The vibration signal collected by the sensor is a comprehensive reflection of the vibration of each component. When the mechanical equipment components such as bearings and gears get fault, the vibration signal will change nonlinearly. Extracting the feature of these changes is the key to realize fault diagnosis. Time-frequency analysis can take into account the global and local aspects of signals and accurately reveal the time-varying characteristics of vibration signals.

The process of time-frequency analysis of vibration signals can be divided into four parts: data acquisition, data preprocessing, time-frequency transform, and data analysis. This paper focuses on the latter two parts of this process, namely time-frequency transform and data analysis, which are keys to establishing the link between vibration signals and fault diagnosis, as shown in Figure 1.

Time-frequency transformation establishes the mapping relationship of vibration signal from time domain to time-frequency domain and uses two-dimensional time-frequency density function to represent the signal, thereby revealing the instantaneous frequency composition of the signal and the time-varying characteristics of each frequency component. The ideal time-frequency transform should have good time-frequency resolution, which means that the signal energy is only concentrated in the instantaneous frequency curve. The structure of mechanical system is complex, and the collected signals are a comprehensive reflection of the vibration signals of each component of the system. The influence of noise and transmission path increases the complexity of signal. Different from the general electrical signal or acoustic signal, the vibration signal of mechanical system often has many subcomponents, each of which is modulated with each other, which makes it difficult to obtain a good resolution of the time-frequency image. Although time-frequency transformation has made great progress in signal processing, many experts and scholars are still making unremitting efforts in vibration signal. The Web of Science database is widely regarded as the standard and most authoritative database for scientific research. The most common 8 time-frequency transformation methods are listed, and according to the search results of Web of Science, the number of relevant papers in the field of mechanical fault diagnosis in the past two decades is counted, as shown in Figure 2. It can be seen that the number of related papers is on the rise as a whole. The early time-frequency transform methods represented by wavelet transform still occupy a large share due to their mature technical characteristics, but in the past ten years, new technical methods have gradually begun to occupy a larger share proportion.

Time-frequency transforms show much richer information about the signal that can reveal the health of machinery. However, due to the complexity of vibration signals of mechanical systems, there are a lot of noise and similar components in the time-frequency image inevitably. How to extract effective information becomes the focus and difficulty of research. Traditionally, the analysis of the information is generally done by engineers with the help of their rich professional knowledge and experience. The earliest analysis method comes from statistics. Through the quantitative analysis of the time-frequency matrix of the vibration signal, the statistical features are extracted as the basis for fault diagnosis. With the continuous improvement of the status of mechanical equipment health operation and maintenance in related fields, the demand for efficiency and accuracy in fault diagnosis is also rising. Lower-level statistical features are often not applicable to complex data structures. Experts and scholars have focused on deeper information hidden in data and proposed many new feature extraction methods. Since the twenty-first century, thanks to the rapid development of the computer industry, mechanical engineering has continued to integrate and develop in the direction of informationize and intelligentize. The combination of machine learning and mechanical fault diagnosis has become a hot topic in the current field. Some achievements have been made in how to effectively use machine learning models to automatically and intelligently analyse the results of time-frequency transformation of signals.

In view of application of time-frequency analysis in fault diagnosis of rotating machinery, Lakis [7] reviewed some time-frequency transformation theories in 2007. Feng et al. [8] gave a more detailed overview of the basic principles, advantages, and disadvantages of various time-frequency transformations since the 1990s and their applications in mechanical fault diagnosis in 2013. The above literature is limited by the year of publication and only covers the research results before 2013. However, in recent years, frequency transformation technology and data analysis technology have developed many new ideas and achieved many achievements in their original fields. It is worth paying attention and trying to combine the two to form a new time-frequency analysis technology to solve the problem of mechanical system fault diagnosis. Many researchers have made such attempts and achieved certain results. These achievements have developed different directions based on different technologies and means, and there is no paper to sort out and summarize these achievements.

In view of this, this paper, as a narrative review, reviews the scientific research achievements of time-frequency analysis in rotating machinery fault diagnosis. It is unrealistic to completely review all relevant articles in the field, so this article focuses on and cites the latest research results in the past 5 years. According to the degree of relevance to the topic discussed in this paper, the author describes and summarizes articles on time-frequency transformation and time-frequency image analysis methods and summarizes the development of related research route.

The remaining of this paper is composed as follows: in the second part, the application of common time-frequency transform methods in rotating machinery fault diagnosis is reviewed, several new time-frequency transform methods in recent years are described, and their ideas and methods are discussed and summarized. The third part reviews the analysis method for time-frequency transform, divides it into three stages, and summarizes the results published, respectively. The fourth part is the conclusion.

2. Time-Frequency Transform

Time-frequency transform technology plays an important role in establishing the link between mechanical vibration signals and their fault diagnosis. To summarize its development process, this paper selects several representative methods to give a general description, among which the formula is only a principle description, and the specific mathematical proof can be found in the references. The citations in this section are limited to papers in the field of rotating machinery fault diagnosis.

2.1. Traditional Time-Frequency Transform

Short-time Fourier transform (STFT) [9] was proposed in 1946 and is the earliest time-frequency transform technique. STFT approximates the instantaneous frequency of time-varying signals by windowing the signal as shown in formula (1), where is the input signal and is the window function, and is a common method for analyzing nonstationary signals. However, due to its simple principle, it is difficult to make theoretical breakthroughs in engineering applications, and it is often used as an auxiliary or basic theoretical tool in the latest research [1012]. The biggest defect of STFT is that it is limited by the uncertainty criterion. Once the window function length is selected, its time resolution and frequency resolution are also determined and cannot meet the requirements of high precision at the same time.

In order to overcome this problem, the wavelet transform (WT) proposed in the 1980s adds time translation and scale scaling parameters to the basis function and has adaptive microscopic ability to frequency changes as shown in formula (2), where Scale controls the expansion of the wavelet function, and translational controls the translation of the wavelet function. Peng and Chu [13] systematically summarized and reviewed the application and development of wavelet transform in the field of mechanical fault diagnosis in 2004. Yan et al. [14] introduced the classical wavelet transform and the second-generation wavelet transform (SGWT) including continuous wavelet transform (CWT), discrete wavelet transform (DWT), and wavelet packet transform (WPT) in 2014.

In recent studies, researchers often combine wavelet transform with other technical methods to enhance its time-frequency representation. Chen et al. [15] proposed an instantaneous time-frequency transform method based on the combination of overcomplete rational extended discrete wavelet transform (ORDWT) and HHT to extract the transient periodic pulse features implicit in vibration signals. Wang et al. [16] proposed an enhanced kurtosis map based on the combination of wavelet envelopes and signal envelopes at different depths to determine the location of the resonance frequency band to realize the identification of weak bearing faults. Wang et al. [17] proposed an adaptive adjustable quality factor wavelet transform (ATQWT) method based on the optimization of time-frequency kurtosis index to realize the separation of weak features in the original signal of bearing early faults. Ma et al. [18] proposed a time-frequency analysis of the synchronous spline kernel chirp wavelet transform (SSCET) by introducing the frequency rotation operator and frequency shift operator of the spline kernel chirp wavelet transform (SCT) based on the synchronous extraction transform, which effectively suppresses the interference of noise on the time-varying characteristics of the vibration signal. Shi et al. [19] proposed a new time-frequency analysis method based on wavelet and synchronous extraction transform, which improved the concentration of time-frequency representation by extracting the frequency spectrum of the wavelet transform of the signal at the corresponding scale of the intermediate frequency. Dai et al. [20] proposed a denoising technique of rotating machinery signals based on the element analysis method and wavelet transform to extract the fault impulse features from the target signal.

Wigner–Ville distribution (WVD) is also one of the earliest time-frequency transformation methods. In recent years, the research direction of scholars still focuses on suppressing its cross-interference term [2124]. However, due to its own principle, the existence of the cross-interference term is inevitable, which makes it difficult to obtain a good time-frequency expression.

Among the traditional time-frequency transforms, the wavelet transform benefits from the energy of its multiresolution analysis and is still widely used in the field when faced with a large number of nonstationary random process signals in practical applications. Although wavelet transform has the requirement of simultaneous detection of high and low frequency parts in the signal, its analysis results depend on the selection of basic functions, and waveforms with low similarity to the basic functions will be ignored.

2.2. Sparse Decomposition

In view of the problem of basis function selection, a sparse decomposition (SD) method was proposed in the 1990s. By constructing a complete dictionary and using a fully redundant function system to replace the orthogonal basis function, the signal can adaptively select the basis function according to its characteristics, and at the same time, the signal can obtain a more concise representation.

For a given signal , it is sparsely represented; i.e.,

Among them, the matrix called dictionary; the vector called sparse coefficient; is the sparse degree of , indicating the number of nonzero elements in .

For the optimal solution of signal sparse decomposition, a large number of experts and scholars have carried out related research, continuously improved the optimization algorithm, and achieved certain results in signal processing such as speech and image. In recent years, some scholars have improved this method and applied it to vibration signals. Huang et al. [25] in 2017 detailed the application of resonance-based sparse signal decomposition (RSSD) in mechanical fault diagnosis. Jin et al. [26] applied the sparse decomposition method based on double Q factor wavelet to the fault feature extraction of wind power planetary gearbox through parameter optimization. Wang et al. [27] established a sparse low-rank decomposition model based on robust principal component analysis for bearing fault vibration signals, revealing the sparsity of fault characteristic frequencies and the low-rank nature of background disturbances. Aiming at the multicomponent selection problem, He et al. [28] proposed a sparse decomposition-based demodulation method for gear-bearing overlapped modulation signals and designed the characteristics of the dictionary matching compound fault of steady-state harmonic atoms and transient shock atoms.

At present, the main application of sparse decomposition is in coding, denoising, and weak signal extraction, and it is rarely used in time-frequency analysis of rotating machinery fault diagnosis because of the hyperparameter assignment problem. And when facing multifeature targets, taking the optimal solution obtained with sparsity as the only constraint may lose the target.

2.3. Empirical Mode Decomposition

Since the twenty-first century, based on the adaptive processing technology of data, empirical mode decomposition (EMD) occupies a pivotal position in the field of nonstationary signal processing. Different from the past time-frequency transformation methods, EMD decomposes the signal according to the time scale characteristics of the data itself and does not need to set any basis function in advance.

If the instantaneous frequency of a signal has physical meaning, then its local signal must be symmetrical with zero mean and has the same number of zero crossings and extreme points. On this basis, Huang et al. proposed the concept of intrinsic mode functions (IMF) [29]. Huang et al. believe that any signal can be split into the sum of several intrinsic modal components. The intrinsic modal components have two constraints:(1)In the entire time range, the number of zero-crossing points and extreme value points differs by at most one(2)At any time, the average value of the local maximum envelope and the local minimum envelope is zero

EMD is the process of decomposing a signal to obtain its intrinsic modal components, as shown in Figure 3. The specific process is as follows:

The mean value of the upper and lower envelopes of the signal is calculated as and calculated the difference between , , and

Judging whether satisfies the IMF constraint conditions, if not, the mean value of the envelopes of is calculated as , calculate the difference.

Repeat above process times until satisfies the IMF constraints. is the first IMF of the signal , which contains the shortest period component in the signal. Obtain the residual

Treat the residual as a new signal and repeat above process.until the termination condition is met

EMD is widely used in the field of mechanical fault diagnosis. In 2013, Lei et al. [30] introduced some applications of EMD algorithm in rotating machinery faults diagnosis based on the classification of diagnostic objects. In recent years, some scholars have further optimized the EMD algorithm from the view of the structure of the EMD algorithm. Zhang et al. [31] optimized the local wave time-frequency analysis method by improving the extremal domain mean mode decomposition (EMMD) and proposing a new screening stop condition, and decomposed the obtained components, and extracted the average instantaneous frequency and energy ratio as the fault. The feature vector constructs a neural network for state judgment. Yan et al. [32] proposed an improved adaptive variational mode decomposition (IAVMD) time-frequency analysis algorithm for rotor fault diagnosis.

At the same time, the researchers also looked at the possibility of combining the EMD algorithm with other algorithms. Amirat et al. [33] proposed a bearing fault detection method based on the combination of empirical mode decomposition and statistical analysis. Liu et al. [34] proposed an integrated time-frequency analysis method based on EMD and WVD and predicted the remaining life of the gearbox through a particle filter method based on the Wiener process state-space model. Wu et al. [35] used convergent empirical mode decomposition (EEMD) to suppress cross-term interference in WVD time-frequency representation of vibration signals and used the compressed local binary mode grayscale histogram as a feature to input into BP neural network to achieve fault classification.

EMD algorithm is based on the iterative process, which is different from the traditional method of decomposing vibration signals using basis functions. It provides a unique idea for subsequent research in terms of time-frequency transformation. However, when the signal contains discontinuous components, EMD will produce frequency aliasing phenomenon [36]. Many scholars have proposed targeted optimization schemes, but it is difficult to fundamentally solve this problem due to the limitation of the recursive-based IMF separation method of EMD.

2.4. Variational Mode Decomposition

In 2014, Dragomiretskiy and Zosso [37] proposed variational mode decomposition (VMD), which achieved nonrecursive IMF decomposition by constructing and solving the variational constraint problem, and effectively avoiding the possible frequency aliasing phenomenon in EMD. In VMD algorithm, the signal is decomposed, assuming that all its IMF components are narrowband signals concentrated near their respective center frequencies, and a constrained optimization problem is established according to the component narrowband conditions, thereby estimating the center frequencies of the IMF components and reconstructing IMF components. The process can be outlined as follows.

IMF is defined as AM and FM signal, does Hilbert transform on IMF component , and finds its analytical signal.

Move the analytical signal to the corresponding baseband by frequency shifting.

Estimate signal bandwidth by H1-Gaussian norm.

To minimize the sum of the bandwidths of each IMF component, the constrained variational model can be expressed as follows:where represents all IMF components and their center frequencies.

The constraint model is solved by the alternating direction method of multipliers, and the center frequency and bandwidth of each IMF component are continuously updated to realize the adaptive decomposition of the signal. Some scholars have successfully applied VMD to mechanical fault diagnosis. Isham et al. [38] briefly summarized the application of VMD in gear and bearing fault diagnosis and proposed that VMD has the disadvantage of parameter-dependent design.

Aiming at this shortcoming, Jiang et al. [39] studied the variation characteristics of the extracted modal center frequency under different initial center frequencies and proposed an icf-guided VMD method to accurately extract the weak damage features of rotating machinery. Li et al. [40] proposed a periodic pulse extraction method based on improved adaptive VMD and adaptive sparse coding shrinkage noise reduction and adaptively determined the number of modes and penalty factors according to different signals for fault diagnosis of rotating machinery.

Some researchers combine VMD with other analysis methods to improve its performance. Chen et al. [41] proposed a variational nonlinear chirp mode decomposition (VNCMD) method, which uses demodulation techniques to convert nonlinear chirp signals into narrowband signals. On this basis, Wei et al. [42] proposed a new variational nonlinear component decomposition (VNCD) method, and by modifying the optimization function of VNCMD, it has stronger adaptive ability in practical applications. Pan et al. [43] proposed a nonlinear sparse mode decomposition (NSMD) method to improve the robustness, which uses local narrowband components obtained by constrained singular local linear operators as the base signal to complete the signal decomposition.

As a newer time-frequency transformation method, VMD is less used in fault diagnosis of rotating machinery at present. Its dependence on parameter design and low computational efficiency provide the possibility for further optimization.

2.5. Synchrosqueezing Transform

In 2011, inspired by the EMD algorithm, Daubechies et al. [44] proposed the synchrosqueezing transform (SST) from the postprocessing of the time-frequency transform. SST can generate a time-frequency matrix with higher aggregation by rearranging the energy distribution. Taking the synchrosqueezing wavelet transform as an example, do wavelet transform with signal and its time-frequency matrix.

Among them, is the scale factor, is the translation factor, and is the wavelet function.

Improving the time-frequency energy aggregation by compressing the value of to . This process can be expressed as follows:where is the scale number of the wavelet.

Similarly, the synchrosqueezing transform method can be used as a postprocessing method for other time-frequency transform methods such as STFT, so as to improve the energy aggregation of the time-frequency matrix.

In the field of mechanical fault diagnosis, Li and Liang [45] proposed a TF method based on generalized synchrosqueezing transform (GST) to detect and diagnose gearbox faults. Feng et al. [46] improved the synchrosqueezing transform using iterative generalized demodulation. Guan et al. [47] designed a velocity synchrosqueezing transform to improve the ambiguity in the analysis of vibration signals of rotating machinery under nonstationary conditions. Ni et al. [48] proposed a time-frequency analysis method based on VMD and SST to improve the problem of low time-frequency resolution when processing multicomponent signals. Yu et al. [4951] proposed an optimization scheme based on fixed point iterative algorithm to solve the problem of energy diffusion in synchrosqueezing transform when processing strong frequency conversion signals and then optimized for the problem of nonredistribution points. Tu et al. [52] proposed the horizontal synchrosqueezing transform, which improved the energy concentration of the time-frequency diagram and achieved the accurate reconstruction of the vibration signal component through the local estimation of the group delay. Yi et al. [53] obtained the high-resolution TF distribution of the gear vibration signal based on the rearrangement method and the second-order synchrosqueezing transform. He et al. [54, 55] proposed the time-rearranged synchrosqueezing transform (TSST), obtained the time-frequency map of the high-energy aggregation by calculating the group delay estimator and rearranging the time-frequency coefficients in the time direction, and at the same time preserved the reversibility. Later, proposed SST based on down-sampling FFT, by combining with selective redistribution and frequency subdivision scheme, balanced accuracy and computational efficiency in the face of large-scale vibration signal processing. Cao et al. [56] improved the computational efficiency of T SST through parameter optimization.

As a postprocessing method, synchrosqueezing transform is excellent in providing high-resolution time-frequency representation by performing energy rearrangement on the basis of traditional time-frequency transform. However, it is sensitive to noise and may get frequency aliasing when analyzing nonstationary signals.

2.6. Synchroextracting Transform

Inspired by the idea of improving the energy aggregation of time-frequency matrix by means of time-frequency rearrangement, Yu et al. [57] proposed the synchroextracting transform (SET). By constructing a synchroextracting operator, the SET has a better antinoise performance. For a single-component signal , its time-frequency obtained by STFT can be expressed as follows:where is the Gaussian window function. By taking the partial derivative with respect to time , the instantaneous frequency trace of can be obtained as follows:

Time-frequency transform expression of SET is defined as

That is, only the frequency components in the range of the instantaneous frequency trajectory are retained, and the frequency components in the rest part are removed so that the time-frequency matrix has high-energy aggregation. is called the synchronous extraction operator. For multicomponent signals, the processing method is the same as the above process when the frequency interval of each component signal is not less than twice the frequency support range of the window function.

On this basis, Chen et al. [58] proposed an improved SET, which has better robustness to noise pollution through the second-order Taylor expansion of the extraction operator. You et al. [59] introduced an improved penalty function based on a convex optimization scheme on the basis of the SET to denoise the signal. Xu et al. [60, 61] proposed a method for extracting and reconstructing synchronously extracted and transformed signal components based on sequential statistical filters and used the shock sensitivity kurtosis index to screen the fault components to realize the fault detection of the inner ring of rolling bearings, and a generalized S-SET is also proposed, which has good robustness to noise. Yu et al. [62] proposed a new technique binding the demodulation technique, and SET is thus in this paper to overcome the limitations that SET technique certain has in dealing with strong time-varying signals. Li et al. [63] presented a new extraction operator to improve the energy concentration of the TFR of a noise contaminated multicomponent signal by using an adaptive ridge curve identification process together with SET to solve the drawback that the time-frequency representation of a signal produced by SET can be affected by noise contained in the signal.

Judging from the current research results, synchronous extraction and transformation have obvious advantages in the resolution of time-frequency images. But at the same time, it also brings the disadvantage of low computational efficiency and can only be used as an offline analysis method.

2.7. Other Methods

In addition to the above time-frequency transformation methods, some scholars have optimized and reorganized the existing methods according to the actual needs of the project and have also achieved certain results. Feng et al. [6466] used Vold-Kalman filtering to separate single-component components in rotating machinery vibration signals and then achieved high-resolution time-frequency representation through methods such as Hilbert transform and high-order energy separation. The frequency content of nonstationary signals provides a solution. Yang et al. [67] proposed a new basis tracking technique, and experiments showed that this method has good sparsity for the time-frequency feature representation of vibration signals. Shi et al. [68] and Huang et al. [69], respectively, proposed a fault diagnosis method combining STFT with generalized demodulation. Chen and Feng [70] proposed an iterative generalized time-frequency redistribution method and improved the time-frequency resolution by decomposing the nonstationary multicomponent signal into a constant-frequency single-component signal. Shi et al. [71] proposed a generalized stepwise demodulation transformation (GSDT) to improve the energy concentration level of time-frequency analysis. Deng et al. [72] proposed a method to automatically determine the optimal ordering of fractional Hilbert transforms using differential evolution (DE) algorithm. Ma et al. [73] used the probability angular velocity algorithm based on the time-frequency representation to construct the phase function of the vibration signal and introduced a generalized demodulation operator to eliminate the rotational speed fluctuation. Guan et al. [74] proposed the velocity synchronous linear chirp transform (VSLCT), which utilizes a time-varying window length to dynamically provide an ideal time-frequency resolution according to changing conditions.

2.8. Summary

Throughout the decades of development of time-frequency transformation technology, it can be found that the traditional time-frequency transformation technology is still widely used, and its theory has become more and more perfect. It can be said that it is difficult to make breakthroughs in the method itself. However, in the past ten years, new time-frequency transform technology based on it has been emerging. Overall, time-frequency transform technology has developed more and more rapidly, and object-oriented has also shifted from analog signals or simple signals to nonlinear time-varying complex signals. Realizing the clear and accurate expression of signals in the time-frequency domain is the main goal of development in this field.

In order to achieve this goal, the current research direction mainly focuses on two aspects: one is to study how to decompose the signal into a single-component modal signal and then adopt Hilbert transform for the single-component modal signal; the other is to study the way of postprocessing and perform energy rearrangement on the time-frequency matrix. Both of these two directions have their own advantages and disadvantages, and each has its own scope of application when facing specific problems as shown in Table 1. It is difficult to compare the advantages and disadvantages of the two qualitatively or quantitatively at present. Moreover, the existing time-frequency transformation methods often bring about a significant reduction in computational efficiency while pursuing high time-frequency resolution, which will also be one of the possible research directions in the future.

3. Data Analysis of Time-Frequency Images

The time-frequency transform shows the richer time-varying characteristics of vibration signals by mapping the one-dimensional time domain signal to the two-dimensional time-frequency plane. However, the time-frequency image does not spontaneously match the fault type, and the relationship between them is often difficult to see intuitively. To establish the connection between the time-varying characteristics in the time-frequency image, the state of the mechanical equipment requires further data analysis. Then, the time-frequency images can be classified according to the results of data analysis. In this section, data analysis methods commonly used in time-frequency analysis of rotating machinery fault diagnosis are divided into three stages and reviewed according to their development history.

3.1. Based on Statistical Method

The earliest data analysis is based on statistical indicators of the data, and such methods are often very intuitive and simple to implement. Zhao et al. [75] proposed to use the contour map to enhance the display of the power distribution in the wavelet transform time-frequency map. Cai Yanping et al. [76] segmented the time-spectrum contour map of the vibration signal according to the image segmentation theory and used fuzzy C-means clustering by selecting the feature parameters such as the centroid position, feature body area, and number and entropy of the segmented image. Li et al. [77] proposed to use the first-order moment of the S-transform time-frequency diagram of the vibration signal as the eigenvector and proved its effectiveness by classifying the vibration signal of the engine under five operating conditions. Dhamande and Chaudhari [78] extracted the time-frequency statistical features from the multilayer wavelet coefficient map of continuous wavelet transform for the problem of bearing composite fault identification for SVM training. Wang et al. [79] constructed a median map based on the video map of nonfaulty data, designed a new metric based on the distance between the time-frequency map of the measured signal and the median map, and used hypothesis testing to make classification decisions.

Although the statistical-based analysis method is simple and efficient, it is greatly affected by the quality of time-frequency images and has poor robustness to noise. It is often difficult to establish a unified quantitative standard for different data sets, and the established diagnostic models lack mobility.

3.2. Based on Feature Extraction

With the continuous progress of industrial engineering, there are further requirements for the accuracy of fault diagnosis of mechanical equipment. It is imperative to deeply mine time-frequency images to reveal their hidden features and establish a diagnostic model with high robustness and generalization. With rich experience and subject knowledge, many scholars have extracted variety of features to guide fault diagnosis from the time-frequency representation of vibration signals.

Some studies have proposed innovative time-frequency image features from the perspective of signal analysis. Zhu et al. [80] used singular value contribution rate and entropy weight to construct multiweight singular value decomposition (MWSVD) and extracted features from the time-frequency matrix obtained by wavelet packet transform. Park et al. [81] proposed a new time-frequency image feature for fault diagnosis of variable speed rotating machinery.

Some studies extract time-frequency image features from the perspective of the image itself. Iatsenko et al. [82] proposed an adaptive ridge extraction method based on dynamic path optimization and fixed point iteration. Huang et al. [83] proposed a multitime-frequency curve extraction method suitable for nonpeak-to-peak ridge lines. Guo et al. [84] proposed a ridge estimation method based on variational nonlinear chirp mode decomposition. Chen et al. [85] proposed a high-precision time-frequency characteristic curve extraction method for variable speed bearing fault diagnosis based on iterative envelope tracking filter (IETF). Shi et al. [86] proposed a linear transformation of frequency matching, which improved the accuracy of frequency ridge extraction in the time-frequency map. Liu et al. [87] proposed a method based on generalized demodulation to iteratively extract time-frequency curves from the time-frequency representation of vibration signal components using a fast path optimization algorithm. Dou and Lin [88] proposed an adaptive variable bandwidth cost function (AVBCF) to adaptively search for ridge extraction regions in time-frequency images. Li et al. [89] proposed an iterative feature ridge extraction (ICRE) strategy to automatically extract multiple feature ridges on the time-frequency plane.

Some studies use numerical methods to analyse time-frequency images. Fu et al. [90] proposed a signal identification method that combines the signal time-frequency matrix with a window to perform singular value decomposition, extracts its eigenvectors, and combines with BP neural network. Cai et al. [91] formed a diagnostic eigenvector by extracting the invariant moment feature of the EMD-WVD vibration spectrum time-frequency image of the vibration signal. Hou et al. [92] extracted the spectral singular values of vibration signals as fault features. Wang et al. [93] proposed a new semisupervised multilayer non-negative matrix factorization method, using a two-layer non-negative matrix factorization model to learn hierarchical attribute representations of fault categories and severity from the time-frequency distribution (TFD) of the signal.

There are also some researches on feature extraction of time-frequency images with the help of technical methods in other fields. Wang Fengtao et al. [94] applied the two-dimensional manifold (2D-LPP) algorithm to effectively extract the time-frequency projection image representing the bearing fault state on the basis of HHT time-frequency analysis. Yang et al. [95, 96] proposed a time-frequency feature extraction method based on matching pursuit technique. Wang et al. [97] used the time-varying autoregression (TVAR) model to decompose the time-spectrogram of the vibration signal, and the threshold value was selected to calculate its energy mean value as a feature. Zhang et al. [98] used the impulse coupled neural network to decompose the optimal generalized S-transform time-frequency map into binary and defined and extracted the capture ratio sequence of the binary image as the characteristic parameter of the rolling bearing fault signal. Yan et al. [99] proposed to use the SVM classification model based on particle swarm optimization to learn the multidomain fault feature information of vibration signals and realize the identification of multifault states of rolling bearings. Ziqian et al. [100] proposed an adaptive dynamic weighted fusion method of time-frequency features based on attention mechanism.

Judging from the research results, with classification accuracy as the standard for extracting time-frequency image features, there are often quite rich mathematical tools that can achieve or approximately achieve the same effect. Based on these tools and relevant experience knowledge in the field of fault diagnosis, many methods of feature extraction have been developed. Despite the rigorous theoretical support, these methods have not yet formed a complete architecture.

3.3. Based on Artificial Intelligence

Although the analysis method based on feature extraction has achieved many achievements in the field, with the vigorous development of the computer industry, in the actual engineering scene, faced with the complex and huge amount of data and the demand for online diagnosis, often requires an automated way to reduce the dependence on engineers. With the help of artificial neural network, it is expected to achieve this goal.

Artificial neural network is an important branch of artificial intelligence. By simulating biological neural network, the computer has the ability to recognize existing knowledge and acquire new knowledge, continuously improve performance, and realize self-improvement [101]. In the past two decades, thanks to the substantial improvement of computer computing power, artificial neural networks have been greatly developed, and their models often have high complexity and large capacity, which means that they can complete more complex learning tasks. Since the twenty-first century, artificial neural networks have demonstrated their superior performance in several tests and competitions. In recent years, artificial neural network models such as deep autoencoders [102, 103], deep belief networks [104], and convolutional neural networks [105, 106] have also been widely used in the field of mechanical fault diagnosis. The basic structure of artificial neural network is shown in Figure 4.

In the early research, scholars often directly applied the neural network model to the classification of time-frequency images. Chen et al. [107] proposed a CNN automatic classification method based on discrete wavelet transform time-frequency map. Wang et al. [108] and Li et al. [109], respectively, proposed a fault diagnosis method using CNN to automatically classify the vibration signal STFT time-frequency diagram. Zeng et al. [110] used CNN to classify the S-transformed time-frequency image of the gearbox vibration signal to realize the identification of fault types. Belmiloud et al. [111] learns with the wavelet packet decomposition of vibration signals as the input data of CNN. Kumar et al. [112] used CNN to learn to classify time-frequency images obtained by wavelet transform of signals in the angle domain. Ma et al. [73] proposed an intelligent fault diagnosis method based on generalized demodulation (GD) with deep residual network (DRN). Wei et al. [113] proposed a fault vibration framework based on residual network (ResNet) and extreme learning machine ELM to classify and identify the time-frequency domain images of vibration signals. Lin et al. [114] proposed an intelligent identification method for gearbox faults that combines variational mode decomposition (VMD) and probabilistic neural network (PNN).

After achieving certain results, many scholars are not satisfied with the simple application of artificial neural network and began to deeply explore its combination with mechanical fault diagnosis. Jing et al. [115] comparatively analyzed the performance of CNN for learning features from raw data and time-frequency domain transformation. Zeng and Li [116] studied the recognition performance of time-frequency matrices constructed by different time-frequency methods of CNN for vibration signals of gearbox transmissions under variable speed conditions. Verstraete et al. [117] validated the ability of the CNN model to accurately diagnose faults by comparing three time-frequency analysis methods as raw signal representations. Shi et al. [118] compared and analyzed the situation when the time-domain grayscale image and time-frequency image of vibration signal were input to CNN from the aspects of model convergence speed and recognition accuracy.

At the same time, some scholars have optimized the mechanical fault diagnosis model based on artificial neural network from different angles with their experience and knowledge in the field.

Starting from the time-frequency matrix as the input of the neural network, Zhu et al. [119] reduced the size of the time-frequency matrix of the vibration signal by bilinear interpolation. Pang et al. [120] transformed the time-frequency map into a low-dimensional encoding matrix with strong sparsity based on the discrete cosine transform (DCT) algorithm to improve the computational efficiency of the CNN model. Wang et al. [121] adopted the newly proposed Bidirectional on-Negative Matrix Factorization (BiNMF) method to obtain the low-rank feature matrix of the time-frequency map, which greatly compresses the computational complexity of the tfr-based prediction algorithm. Wan et al. [122] used convolutional autoencoder (DCAE) to perform image denoising on wavelet transform time-frequency maps of vibration signals in different states to improve the classification effect of CNN. Drawing on the practice of CNN model in image processing, Hasan et al. [123] performed grayscale image processing on the continuous wavelet transform time-frequency map of vibration signal and then sent it to the network for pattern recognition, which improved the accuracy of network recognition. Zhao et al. [124] used the synchronous extraction transform to eliminate the divergent energy in the time-frequency image, which improved the network’s ability to identify fault features in the time-frequency image in the case of strong noise.

For the characteristics of vibration signals, starting from the optimization of the network structure, Jahromi et al. [125] optimized the dynamic fuzzy neural network based on sequential fuzzy clustering with the input of the time-frequency map of the dimensionality-reduced wavelet transform. He and He [126] embedded a time-synchronized resampling mechanism in the deep learning structure to introduce vibration analysis as a hard constraint into the deep learning structure, resulting in higher diagnostic accuracy. Liu et al. [127] proposed a fault diagnosis method based on an improved multiscale residual generative adversarial network and a feature-augmented-driven capsule network. The adversarial generative strategy was used to effectively solve the problem of imbalance between fault data and nonfault data in intelligent bearing diagnosis. Ding et al. [128] proposed a new time-frequency transformer model based on the Transformer model for extracting abstract features from video representations of vibration signals. Bearing experimental data verify its feasibility.

In addition, some scholars also put forward optimization ideas from other aspects. Facing the problem of possible missing samples in fault diagnosis, Zhang et al. [129] proposed an improved convolutional neural network (CNN) method, which utilizes multiple parallel convolutional layers to achieve complementary time-frequency feature extraction. Facing the computational pressure brought by the high sampling rate of vibration signals, Huang et al. [130] adopted the channel attention deep residual network (CADRN) as a diagnostic model and proposed a time-frequency fault diagnosis method for planetary gearboxes in a cloud environment.

3.4. Summary

This section reviews the development of data analysis for time-frequency matrices of vibration signals, and divides it into three stages: based on statistics, based on feature extraction, and based on artificial intelligence as shown in Table 2. It should be noted that the stage division here is carried out according to the order of publication, but it does not mean that they have a substitution relationship with each other. The statistics method extracts the surface information of the time-frequency matrix, so it is often easily disturbed by components such as noise. The feature extraction method generally has good robustness by mining the deep information contained in the time-frequency matrix, but at the same time, it relies on the experience knowledge of experts and scholars in related fields. The artificial intelligence method reduces the need for experience to a certain extent by constructing an end-to-end artificial neural network diagnostic model, and the diagnostic model adaptively extracts features from the time-frequency matrix and establishes a connection with the mechanical health status. However, it is not enough to apply the artificial neural network model to mechanical fault diagnosis according to the prior knowledge of computer and other fields. The optimization of the model is still inseparable from the experience and knowledge of experts and scholars in the field.

Predictably, with the rapid development of computer technology and the overall popularization of artificial intelligence algorithm, intelligent and automatic data analysis method will play a dominant role in mechanical system fault diagnosis. However, the “black box” problem of artificial intelligence algorithm will be the biggest obstacle to theoretical research and practical application. More and more scholars pay attention to this problem, how to enhance the credibility of intelligent algorithms through expert knowledge will be the future research trend.

4. Conclusions

This paper reviews the application of time-frequency analysis in mechanical fault diagnosis and divides it into two parts: obtaining time-frequency images and extracting features from time-frequency images. In terms of obtaining time-frequency images, in order to obtain a time-frequency matrix with higher resolution and energy aggregation, the current research directions of experts and scholars mainly focus on the single-modal component decomposition of the signal and the energy rearrangement of the time-frequency matrix. In terms of feature extraction for time-frequency images, statistical-based feature extraction methods can only extract surface information; feature extraction-based analysis methods can mine deeper information in time-frequency matrices but rely heavily on the experience and knowledge of experts and scholars. The analysis method of artificial intelligence builds an end-to-end artificial neural network diagnosis model between the time-frequency matrix and the mechanical state and automatically learns and extracts features from the time-frequency matrix. However, in the application of artificial neural network models, the complexity of the models involved is very high, so that better performance can often be achieved only by adjusting the parameters, and the requirements for users are low, but the model lacks a strict theoretical basis [131]. In the long run, such an approach often creates hidden dangers because the reliability of the model cannot be assessed. From a scientific point of view, it is highly inadvisable to rely on luck to construct and tune deep learning network parameters. The optimization of the model and the evaluation of its reliability must return to the empirical knowledge of experts and scholars. The heavy workload of extracting features from big data is adaptively completed by the artificial neural network model, and it should be the future development direction to analyse and evaluate the situation of feature extraction based on empirical knowledge to make the final decision.

Among the methods of time-frequency analysis, time-frequency transformation and data analysis are two parts of complementary and parallel development. The forward direction of time-frequency conversion technology is to find better time-frequency representation of vibration signals to fully display signal characteristics. The research of data analysis technology focuses on how to mine features and establish a direct mapping relationship between features and fault types. In the current study, these two parts tend to be independent of each other. How to combine the two more effectively may be one of the future research directions. In addition, although the introduction of intelligent algorithms has improved the effectiveness and efficiency of time-frequency analysis, it has led to the decline of credibility to some extent. The choice of intelligence, automation, and reliability, or to achieve a certain balance state, will also be the direction worth exploring in the future research.

Conflicts of Interest

The authors declare that there are no conflicts of interest.

Acknowledgments

This work was supported by the Key R&D program of Shanxi Province (International Cooperation, Grant no. 201903D421008) and the Beijing Municipal Natural Science Foundation (Grant no. 3192025).