Relaxation time due to electron-longitudinal-acoustic (LA) phonon interaction is calculated in a GaAs quantum dot (QD) with N electrons (from N = 1 to 4), where electrons in a narrow quantum well are confined by a parabolic confining potential, by using the exact eigen states of electrons. Although the energy levels become dense with increasing the number of electrons, the modification of the relaxation time is found to be not so strong, which attributes the fact that many electron eigen states consist of only a few dominant single electron states which limit the electron relaxation. By comparing relaxation process via intermediate states with the direct process, several fastest processes are found to be realized by relaxation through intermediate states between the initial state and the ground state. The effect of change in the quantum well width is also discussed.