This paper describes a vectorial finite element method for the analysis of electromagnetic waves propagating in helical systems. The present method can be as easily used as the methods for guided electromagnetic waves in systems with translational and axial symmetries. The functional for the vector Helmholtz equation is minimized using twisted coordinates in which the problem becomes two‐dimensional. The spurious solutions, which do not satisfy the divergence‐free condition, are removed on the basis of the penalty method. Two finite element equations for the covariant and contravariant components of electromagnetic fields are derived and boundary conditions for both formulations are described. The present method is applied to the analysis of a twisted waveguide with a rectangular cross section and the numerical solutions are shown to be in good agreement with the perturbed solutions for a small helical pitch.