A small signal theory of a free‐electron laser (FEL) with a rotating electron beam in a uniform axial magnetic field and in an azimuthal wiggler field (the ‘‘circular’’ FEL) is developed. The analysis includes the low and high gain regimes and the influence of longitudinal space‐charge forces. It is found that the circular FEL instability has two regimes: a strong pump regime and a negative mass dominated regime. The negative mass regime replaces the weak pump (Raman) regime found for the usual FEL geometry in which the electron beam propagates in the axial direction (the ‘‘linear’’ FEL). The dispersion relation is evaluated, and the resulting growth rates are compared with those of the linear FEL. For a cold beam, at fixed output frequency, the growth rate in the strong pump regime is larger, by a factor of &ggr;2/3, in the circular FEL. The negative mass instability is shown to increase the growth rate and modify the bandwidth of the circular FEL in the strong pump regime. However, the circular FEL performance is found to be more sensitive to the energy spread than the linear FEL.