A self‐consistent theory of the free electron laser instability is developed for a hollow electron beam propagating through an undulator (multiple mirror) magnetic field. The stability analysis is carried out within the framework of the linearized Vlasov–Maxwell equations. It is assumed that the beam is thin, with radial thickness much smaller than the mean beam radius, and that &ngr;/&ggr;b≪1, where &ngr; is Budker’s parameter and &ggr;bmc2is the characteristic energy of the electron beam. The dispersion relation describing the free electron laser instability in a hollow relativistic electron beam is obtained for an equilibrium distribution function in which all electrons have the same value of transverse energy and the same value of canonical angular momentum, and a Lorentzian distribution in axial momentum. It is shown that the influence of finite radial geometry plays a critical role in determining the detailed stability behavior. Moreover, the growth rate and bandwidth of the instability can be expressed in terms of Budker’s parameter &ngr;, instead of the plasma frequency as in the case of a uniform density beam. Furthermore, it is found that free electron laser stability properties exhibit a sensitive dependence on axial momentum spread.