Modulational instabilities of electromagnetic electron cyclotron waves in a dense magnetized plasma