Driving force for a nonequilibrium phase transition in three-dimensional complex plasmas