When electron cyclotron resonance (ECR) plasma heating is operated at a frequency exceeding the minimum electron cyclotron frequency in an open trap, the power deposition zone is shifted from the trap center towards higher magnetic fields. In this case, ECR heating may result in the formation of 'butterfly-like' distribution functions of fast electrons characterized by holes in a velocity space formed by particles that can not reach the heating zone while moving adiabatically from the trap center. In this paper, we develop a basic three-dimensional (1d-real + 2d-velocity space) kinetic model that allows us to describe the effects of the butterfly-like distributions on the whistler wave instability. We calculate a linear amplification gain for a whistler wave propagating along the magnetic field in an essentially inhomogeneous plasma, and compare the results with the well-studied case of anisotropy-driven whistler instability. The proposed theory is primarily aimed at the interpretation of recent experimental data accumulated in laboratory studies of kinetic instabilities and stimulated emission of ECR plasma discharges in mirror traps.