With the channeling technique the lattice location of both As and B is studied in single As-or B-doped and in doubly As-and B-doped silicon single crystals. The influence of the position of the Fermi-level on the displacement of impurity atoms off substitutional lattice sites is investigated by changing the crystals fromn-top−type or vice versa by choosing implant conditions and annealing termperatures for the doubly doped crystals in an appropriate way. Big changes were found in displacement cross sections for As and B after conversion of the crystals fromn-top-type. The results can be explained by assuming that the interaction between primary defects and impurity atoms causing the displacement of the impurity atom is controlled by Coulomb attraction between charged point defects and the impurity atoms.