During his long and illustrious career, Professor Kiritani made many of the most significant and revealing observations regarding the nature of the primary damage state and the fate of the produced defects in irradiated metals and semiconductors. We present a review of recent results of molecular dynamics (MD) and kinetic Monte Carlo (KMC) simulations of defect production and annealing in irradiated metals and semiconductors. The MD simulations describe the primary damage state in two prototypical elemental metals and in one-elemental semiconductor, namely Fe, Au, and Si. These materials were all thoroughly investigated by Prof. Kiritani and his colleagues using neutron irradiation followed by TEM observation, and here we attempt to provide some further understanding of the experimental observations by using atomic-scale computer simulation tools. We describe the production of interstitial and vacancy clusters in the cascades and highlight the differences among the various materials. In particular, we discuss how covalent bonding in Si effects defect production and amorphization resulting in a very different primary damage state from the metals. We also use MD simulations to extract prefactors and activation energies for migration of point defects, as well as to investigate the energetics, geometry and diffusivity of small vacancy and interstitial clusters. We show that in the metals, small interstitial clusters are highly mobile and glide in one dimension along the direction of the Burgers vector. The results for the primary damage state and for the defect energetics and kinetics are then combined and used in a KMC simulation to investigate the escape efficiency of defects from their nascent cascade in metals. We show that in fcc metals Au and Pb at or above stage V the escape probability is approximately 40% for 30 keV recoils so that the freely migrating defect fraction is approximately 10% of the displacement per atom (dpa) standard.