An efficient approach to Structural Health Monitoring of dynamical systems based on the Wavelet Transform (WT) and the concept of subspace angle is presented. The objective is to propose a detection method that is sensitive to the onset of nonlinear behaviour in a dynamic system. For this purpose, instantaneous frequencies are identified first from output-only vibration signals using the Wavelet Transform. Time varying deformation shapes are then extracted by analyzing the whole measurement data set on the structure. From this information, different dynamic states of the structure may be detected by inspecting time variations of 'modal' features. The experimental structure considered here as application example is a clamped beam with a geometric nonlinearity. Detection of nonlinearity is carried out by means of the concept of subspace angles between instantaneous deformation modes extracted from measurement data using the continuous Wavelet Transform. The method consists in controlling the angular coherence between active subspaces of the current and reference states respectively. The proposed technique, which shows a good sensitivity to small changes in the dynamic behaviour of the structure, may also be used for damage detection.