The effect of the finite axial magnetic field on the excitation of a backward wave oscillator (BWO) is investigated. The driver beam is assumed to be mono‐energetic helical electron beam such that all the constituent electrons have their gyration centers on the axis of the slow wave structure (SWS). Such a beam supports negative energy fast and slow cyclotron modes (FCM and SCM) that can excite the structure modes in the SWS. This may contribute to the microwave generation in BWO. All the previous analyses on BWO have assumed the electrons without the initial perpendicular velocity component. The formulation is as follows: The thin annular large orbit beam is perturbed and the first order perturbations in velocity and density are obtained. Integrating radially across the beam, we derive the expression for surface current density. It can be expressed in terms of the azimuthal and axial components of the perturbed electric field. The boundary conditions on the beam surface are as follows. (a) Matching of the continuous axial and azimuthal components of electric field and (b) Matching of the discontinuous axial and azimuthal magnetic fields across the beam by the presence of surface current density. They are augmented by the requirement of Floquet periodicity on the RF fields and the boundary conditions that the tangential electric field should vanish on the metal SWS surface, 6(2N+1) × 6(2N+1) order determinant that must to be zero and this is the dispersion relation of the system. Here, 2N+1 is the number of Floquet harmonics involved. Numerical analysis is made assuming N=4 and appropriate practical experimental parameters. The excitations of the unstable non‐axisymmetric Q‐TE11and Q‐TM11modes caused by negative energy FCM in addition to unstable axisymmetric Q‐TM01mode caused by slow beam mode that is identical to conventional BWOs are observed. Defining &agr; as the ratio of the transverse to the longitudinal velocity components of the beam electrons, it is found that for &agr; < 0.065 the growth rate of the conventional axisymmetric Q‐TM01mode is dominant, while for &agr; ⩾ 0.065 the growth rate of non‐axisymmetric Q‐TE11mode dominates for the chosen set of parameters. The effect of slow cyclotron instability is comparatively small. © 2003 American Institute of Physics