Growth rates of negative mass instabilities in conventional and modified betatrons are calculated by analytic methods and by performing three‐dimensional particle simulations. In contrast to earlier work, toroidal corrections to the field equations are included in the analytic model. As a result, good agreement with numerical simulations is obtained. The simulations show that the nonlinear development of the instabilities can seriously disrupt the beam.