This article describes and illustrates a local instability that may arise when using the method of least median squares (LMS) to fit models to data. This idea is contrary to the generally held belief that the method is highly resistant to perturbations in the data. In fact, slight changes in centrally located data can cause the LMS estimate to change by a large amount. The LMS method uses a criterion function that is calculated on half samples. If there are two or more half-samples with roughly the same value of the criterion function, then by slight changes in some of the data the LMS solution can be made to jump from one half sample to the other. An example of a data set from a standard text that exhibits this feature is presented. This suggests that some caution should be exercised when using this method. It does not automatically guarantee complete robustness to misspecifications in the data.