Recent experiments have demonstrated that a high separatrix density and a large ratio of separatrix density to pedestal top density are two crucial conditions for achieving high confinement operation with small edge localized modes (ELMs). In order to identify the underlying physics of this phenomenon, a series of equilibria with different separatrix and pedestal top densities are constructed, and their peeling–ballooning (P–B) instabilities are analyzed through simulation. It is found that there is a threshold value of pedestal top density which comes from competition between ion inertia and diamagnetic effect, and ELM energy loss can be minimized at the threshold value for a fixed separatrix density. When the pedestal top density is smaller than the threshold value, the ion inertial effect induced by the density profile has a significant influence on the growth of ELMs, resulting in an increased linear growth rate and more ELM energy loss by trigging low-n modes (n being the toroidal mode number) in the nonlinear phase. When the pedestal top density is larger than the threshold value, the diamagnetic effect is the main factor determining the mode spectrum, which moves to the high-n region with a larger growth rate and the nonlinear ELM energy loss increases. However, for a fixed pedestal top density, a larger separatrix density leads to a wider mode spectrum with a smaller growth rate; thus ELM energy loss is reduced. The results of this research provide a new mechanism, namely that the P–B mode is possibly transferred to a resistive ballooning mode, to interpret the experimental findings during high pedestal density operation.