Defects on the surface of railroad tracks have been the cause of growing concern over the past three decades. The automated detection and classification of rail surface defects would be of great assistance to rail maintenance planners, who develop grinding strategies to prevent the development of potentially dangerous deterioration. Videotaped images of the surface of rail have been obtained, but they are subject to distortions due to the acquisition process as well as physical phenomena on the track itself. In this analysis, an algorithm is presented for the simultaneous restoration and segmentation of objects in a two-dimensional image. The algorithm relies on distributions that model the relationships between sites and neighbors in order to restore a distorted image to an estimate of its ideal form, and also obtain detailed information about the objects located in the image. The foundation of the algorithm is the Iterated Conditional Modes procedure for image restoration. The resulting extension is capable of providing detailed measurements of the geometric features of objects detected in an image. The extended algorithm is applied to an image distorted by simulated noise, and also to an image taken from a videotape of a rail surface. The results of the analysis demonstrate the potential for accurate detection, measurement, and classification of rail surface defects.