New edge diagnostics and detailed analysis of magnetic topology have significantly improved the comprehension of the processes developing at the boundary of a reversed-field pinch (RFP) plasma in RFX-mod (a = 0.46 m, R = 2 m).An upper critical density nC ≈ 0.4 nG (nG Greenwald density) is found to limit the operational space for the improved quasi-single helical (QSH) regime: magnetic topology reconstructions and diagnostic observations suggest that this limit is due to a helical plasma–wall interaction which determines toroidally and poloidally localized edge density accumulation and cooling.The experimental evidence is provided by a variety of diagnostics: the magnetic boundary as reconstructed from equilibrium codes reveals a helical deformation, which is well correlated with the modulation of edge pressure profile as reconstructed from the thermal helium beam diagnostic. Correlations with the helical deformation are also observed on the space- and time-resolved patterns of the floating potential measured at the wall, and with the edge plasma flow, obtained from different diagnostics. The relevance of these findings is that understanding the mechanisms that limit the operational space of QSH is decisive in achieving the goal of high-density stationary helical RFP equilibrium.