The Lyapunov exponent has been investigated both numerically and analytically for various plasma states. With the use of a three-dimensional particle code that calculates Coulomb forces among individual particles, instantaneous expansion rates of nearby trajectories in a phase space are studied. It is found that the Lyapunov exponent has different dependences on the Coulomb coupling constant corresponding to dilute, liquid and solid plasmas. An analytical model is also developed and it gives a relation between the Lyapunov exponent and the dielectric response function. The model indicates that the Lyapunov exponent is of the order of the plasma frequency in a dilute plasma, which agrees with simulation results. In a dilute plasma, large fluctuation of the instantaneous expansion rate is shown to be caused by a close encounter which breaks the symmetry of force and leads to trajectory instability.