AbstractThis paper calculates the reflection and transmission coefficient at a discontinuity in rectangular dielectric waveguides. At the discontinuity, waveguides of two different sizes are connected concentrically. Generally speaking, radiation modes generated at the discontinuity in open waveguides make analysis difficult.Here, double Fourier transformation is applied to field distributions at the discontinuity so that a guided and radiation mode can be treated identically. Since the reflection and transmission coefficient can not be obtained immediately by solving the boundary condition equations at the discontinuity, a ratio of electric and magnetic field intensities of the radiation mode is introduced and converged to constant by an iterative approximation. This approximation formula is derived from an orthogonal relation between the guided and radiation mode. If this ratio is converged to a steady value, the reflection and transmission coefficients are converged simultaneously. In addition, the reflection, transmission and radiation power at the discontinuity are calculated numerically. These results are compared with those obtained from the mode matching method which is useful when the discontinuity is so small.The theory in this paper is advantageous in analyzing the discontinuity at a waveguide junction. The analytical method is applicable to an off‐axial junction of the waveguide and the analysis of a rectangular dielectric rod antenn