Use is made of the single‐particle orbit equations together with Maxwell’s equations and appropriate statistical averages to investigate detailed properties of the sideband instability for a helical‐wiggler free‐electron laser with wiggler wavelength &lgr;0=2&pgr;/k0=const and normalized wiggler amplitudeaw=eBˆw/mc2k0=const. The model describes the nonlinear evolution of a right‐circularly polarized primary electromagnetic wave with frequency &ohgr;s, wavenumberks, and slowly varying amplitudeaˆs(z,t) and phase &dgr;s(z,t) (eikonal approximation). The orbit and wave equations are analyzed in the ponderomotive frame (‘‘primed’’ variables) moving with velocityvp=&ohgr;s/(ks+k0) relative to the laboratory. Detailed properties of the sideband instability are investigated for small‐amplitude perturbations about a quasisteady equilibrium state characterized byaˆ0s=const (independent ofz’andt’). Two cases are treated. The first case assumes constant equilibrium wave phase &dgr;0s=const, which requires (for self‐consistency) both untrapped‐ and trapped‐electron populations satisfying ⟨∑j exp[ik’pzj0(t’) +i&dgr;0s]/&ggr;’j⟩=0.Herekp=(ks+k0)/&ggr;pis the wavenumber of the ponderomotive potential,z’j0(t’) is the equilibrium orbit, and &ggr;jmc2is the electron energy. The second case assumes that all of the electrons are deeply trapped, which requires a slow spatial variation of the equilibrium wave phase, ∂&dgr;0s/∂z’=2&Ggr;0(&Ggr;0ck0/&OHgr;B)2k’p≠0. The resulting dispersion relations and detailed stability properties are found to be quite different in the two cases. Both the weak‐pump and strong‐pump regimes are considered.