A self-consistent nonlinear theory of a relativistic electron beam propagating through a waveguide loaded with a sheath helix is developed by making use of Maxwell equations. A closed integrodifferential equation is obtained for the normalized beam current described in terms of the normalized time &thgr; and propagation distance &zgr;. An analytical investigation of the partial integrodifferential equation of the current modulation is carried out by Fourier decomposing the current modulation with harmonic numberl. The mode strengthcl(&zgr;)is obtained in terms of the spatial oscillation frequency&eegr;land growth rate&xgr;lof the mode amplitude, which are, in turn, determined in terms of the electrostatic and helix effects. The spatial oscillation frequency increases as the electrostatic effect(gl)increases. On the other hand, the growth rate increases with the strength of the helix effect. It is found that the mode strengthclis oscillatory and that it grows during propagation when the spatial frequency&eegr;l(the electrostatic effect) is considerably larger than the growth rate&xgr;l(the helix effect). Otherwise, the exponential growth dominates. Investigation of the helix mode must include the electrostatic effect for an intense beam.