Dispersion relations of nonlinearly coupled electromagnetic and electrostatic waves in relativistic plasmas