The initial stages of molecular beam epitaxy of GaSb on highly mismatched GaAs(001) substrates were investigated. Transmission electron microscopy was used to analyze the defect structure in GaSb islands and at their interfaces with GaAs(001) at different stages of growth. Based on experimental observations, we propose that the semiperiodic net of 90° misfit dislocations at the GaSb/GaAs(001) interface nucleate homogeneously at the leading edges of advancing {111} planes. After nucleation, they glide inwards along the interface plane to reach their equilibrium position. Threading dislocations in GaSb layers were directly correlated with the misfit dislocation net. We demonstrate that there are no threading defects in GaSb islands when their interfaces consist solely of 90° misfit dislocations, and that threading dislocations in the GaSb epilayer are all associated with minority 60° misfit dislocations nucleated in growing islands. The number of threading dislocations per unit area of the GaSb film is found to be independent of GaSb coverage, indicating that island coalescence does not substantially increase the number of 60° dislocations. ©1997 American Institute of Physics.