The effect of impurities on the annealing behavior of irradiated silicon was studied through an investigation of isothermal annealing of minority carrier lifetime in silicon crystals containing phosphorus, arsenic, antimony, or bismuth in the temperature range 100°–180°C. The activation energy for the annealing of vacancy‐impurity complex increased with increasing atom size of the dopant. The values are 0.93, 1.27, 1.84, and 2.22 eV. The frequency factor was also found to be dependent on impurity as well as concentration of the complex. The variation of these parameters is discussed in terms of the lattice strain associated with impurity atoms which have a larger covalent radius than silicon.