A free electron laser instability is investigated for a relativistic annular electron beam propagating through a helical wiggler magnetic field. It is assumed that the beam is thin, with radial thickness (2a) much smaller than the beam radius (R0), and that &ngr;/&ggr;b≪1, where &ngr; is Budker’s parameter. The stability analysis is carried out within the framework of the linearized Vlasov–Maxwell equations for perturbations with general azimuthal harmonic numberland radial mode numbers, including the important influence of (a) finite beam geometry in the radial direction, (b) positioning of the beam radius relative to the outer conducting wall (R0/Rc), and (c) finite wiggler amplitude (&dgr;B). All of these effects are shown to have an important influence on stability behavior. Moreover, the maximum coupling between the transverse and longitudinal modes increases substantially with increasing radial mode numbers. It is also found that the transverse magnetic (TM) mode has slightly larger growth rate than the transverse electric (TE) mode.