Effect of beam emittance on self-modulation of long beams in plasma wakefield accelerators