This paper reports on the results of an experimental investigation into electron cyclotron resonance heating (ECRH) using the fundamental O-mode in the large helical device (LHD). The aim of the experiment is to understand the mechanism of the selective excitation of the O-mode EC wave and, thereby, optimize the performance of the ECRH. The polarization angle and the ellipticity of the electric field of the incident high power millimetre waves were varied in the experiment using a pair of grating polarizers. The responses of the plasma stored energy and electron temperature were measured to evaluate the effects. The optimum values of the polarization angle and the ellipticity for the ECRH were identified experimentally. A simple model of power branching between the O- and X-modes is introduced in order to interpret the observed experimental results. Specific to the ECRH in the LHD configuration, the effect of magnetic shear in the plasma peripheral region is considered.