A possible correlation between the annealing of secondary defects in silicon with high P background and anomalous diffusion of As or Sb was investigated. The P background (9×1019–4×1020cm−3) was formed by phosphorus implantation (80 keV (2–10)×1015cm−2), followed by rapid thermal annealing. Dislocation loops and misfit dislocations of various densities were formed under these conditions. Subsequently, As or Sb was implanted, and finally the crystals were annealed by rapid thermal annealing. By a combination of Rutherford backscattering/channeling, transmission electron microscopy and Hall measurements, anomalous diffusion of Sb and As and reduction or complete annealing of the secondary defects were found in the presence of the high P background. The results are discussed in terms of interaction between impurities and dislocations or defect complexes.