Torsional vibration responses of a nonlinear shafting system are studied by a modified Riccati torsional transfer matrix combining with the Newmark-β method. Firstly, the system is modeled as a chain consisting of an elastic spring with concentrated mass points, from which a multi-segment lumped mass model is established. Secondly, accumulated errors are eliminated from the eigenfrequencies and responses of the system's torsional vibration by this newly developed procedure. The incremental transfer matrix method, combining the modified Riccati torsional transfer matrix with Newmark-β method, is further applied to solve the dynamical equations for the torsional vibration of the nonlinear shafting system. Lastly, the shafting system of a turbine-generator is employed as an illustrating example, and simulation analysis has been performed on the transient responses of the shaft's torsional vibrations during typical power network disturbances, such as three-phase short circuit, two-phase short circuit and asynchronous juxtaposition. The results validate the present method and are instructive for the design of a turbo-generator shaft.