A new helical system ('modular heliotron') with improved modular coils compatible with an efficient closed helical divertor and a good plasma confinement property is proposed, based on a heliotron system with continuous helical coils and one pair of poloidal coils. The physics optimization of this system as a function of the gap angle between adjacent modular coils has been carried out by means of vacuum magnetic surface calculations and finite-beta plasma analyses, and a new improved coil system is invented by combining sectored helical field coils with sectored returning poloidal field coils. A modular heliotron with standard coil winding law (the reference modular heliotron) was previously proposed, but it is found that this was not appropriate to keep a clean helical divertor and high beta configuration when the coil gap becomes large. By modulating the modular coil winding with outside-plus and inside-minus pitch modulation, almost the same good magnetic configuration as that of a conventional heliotron can be produced. The optimal gap angle is determined as a function of the modulation parameter. This improved modular heliotron permits a larger gap angle between adjacent modules and produces a cleaner helical divertor configuration than the reference modular heliotron. All these helical systems are created by only modular coils without poloidal coils