Structural measurements of neutron and ion damage annealing in Si and Ge are compared and interpreted in terms of the annealing of identified defects. Either or both of two prominent isochronal annealing stages above room temperature in neutron irradiated Si are exhibited by X-ray, electron microscope, infrared, and electrical measurements. It is shown that the same two annealing stages are observed in Rutherford scattering measurements of ion implanted Si. Two similar prominent annealing stages are also observed for structural measurements of neutron and ion irradiated Ge. For both Ge and Si the center temperatures for the two annealing stages are approximately 0.35 and 0.5 times the melting temperature. The annealing kinetics and activation energies for the lower temperature stage in Si are correlated with the kinetics and activation energies obtained by analyzing previously reported results on the effects of the substrate temperature and the ion irradiation rate. From a comparison of the time and temperature dependences of the annealing of electron, neutron, and ion damage in Si, it is concluded that the lower temperature isochronal annealing stage near 200°C and the annealing that governs the dose rate dependence of the crystalline to amorphous transition is a consequence of the annealing of divacancies. Support for this conclusion is given by recent infrared measurements of the annealing of ion produced divacancies in Si. The high temperature stage is shown to be near the temperature observed for the epitaxial growth of Si on Si crystal substrates and is interpreted to be related to the annealing of amorphous Si.