A quasilinear model is developed that describes the nonlinear evolution and stabilization of the free electron laser instability in circumstances where a broad spectrum of waves is excited. The relativistic electron beam propagates perpendicular to a helical wiggler magnetic fieldB0=−Bˆ cos k0 z eˆx−Bˆ sin k0 z eˆy, and the analysis is based on the Vlasov–Maxwell equations assuming ∂/∂x=0=∂/∂ yand a sufficiently tenuous beam that the Compton‐regime approximation is valid (&dgr;&fgr;&bartil;0). Coupled kinetic equations are derived that describe the evolution of the average distribution functionG0( pz,t) and spectral energy densityEk(t) in the amplifying electromagnetic field perturbations. A thorough exposition of the theoretical model and general quasilinear formalism is presented, and the stabilization process is examined in detail for weak resonant instability with small temporal growth rate &ggr;ksatisfying ‖&ggr;k/&ohgr;k‖≪1 and ‖&ggr;k/k &Dgr;vz‖≪1. Assuming that the beam electrons have small fractional momentum spread (&Dgr; pz/p0≪1), the process of quasilinear stabilization by plateau formation in the resonant region of velocity space (&ohgr;k−kvz=0) is investigated, including estimates of the saturated field energy, efficiency of radiation generation, etc.