We present a quantum-relativistic model for the nonlinear interaction between large-amplitude electromagnetic (EM) waves and a quantum plasma. The model is based on a collective Klein–Gordon equation for the relativistic electrons, which is coupled with the Maxwell equations for the EM and electrostatic fields. The model is used to derive a nonlinear dispersion relation for the EM scattering instabilities in a relativistic quantum plasma. With application to the quantum free-electron laser (FEL), a relativistic electron beam is passing through an intense EM wave used as a wiggler to produce coherent tunable radiation. The frequency upshift of the radiation is tuned by the energy of the electron beam. The nonlinear dispersion relation reveals the importance of quantum recoil effects and oblique scattering of the radiation on the gain of the quantum FEL.