As NDE simulation tools begin to find more widespread use in critical engineering applications, it is becoming increasingly important to quantify their accuracy. The long‐range goal of simulator development is to develop a suite of tools that can be used throughout the world to predict the results of NDE measurements. However, in many simulations, various approximations are made to speed the computations or increase flexibility and user friendliness. It is therefore necessary to define the range of conditions over which particular models can be used with confidence. This paper summarizes an ultrasonic model benchmarking activity that is being conducted by the World Federation of NDE Centers. The first two phases of this activity involved the comparison of the predictions of a series of models for a set of benchmark problems involving well‐defined experimental geometries and input signals. Those results will be reviewed. In many cases, the models were found to be in good agreement. However, regions were identified that require further attention. Lessons learned will be summarized and actions that have been taken will be discussed. The final step in qualifying a simulation tool for industrial use is a validation against experimental data. Plans for a set of validation experiments, to be conducted during coming years, will be summarized. © 2004 American Institute of Physics