Longitudinal oscillations have been excited in debunched beams in several of the Fermilab synchrotrons, which indicate, under certain conditions, the occurrence of a variety of nonlinear coherent phenomena. These phenomena indicate that mode saturation and longitudinal emittance growth may be influenced by classical processes such as three‐wave coupling and wave overturning due to self‐trapping. The purpose of these studies was to determine the intrinsic machine impedance and the response of the beam to longitudinal perturbations using standard beam‐transfer function measurements. However, experimental observations indicated that the expected linear response regime was absent over a wide range of experimental conditions, consistent with the fact that these machines are longitudinally near marginal stability. The beam response was characterized by resonant three‐wave mode‐mode coupling and by a slow echo‐like phenomenon suggesting phase space rotation. A theoretical investigation was undertaken to ascertain the nature of this response. Using a well‐known perturbative approach, nonlinear coupling thresholds were derived for the onset of the three‐wave interaction and were found to be consistent with measured conditions in the beam. In addition, particle simulations were carried out to attempt to explain mode saturation effects and the long‐time behavior. The results indicate that a temporal mode cascade can occur as a consequence of the three‐wave interaction. Also, self‐trapping in an impedance‐induced potential well, accompanied by phase rotation, may take place, and this effect appears to be associated with pump depletion of the driving mode in the three‐wave interaction. These phenomena play a key role in determining the emittance growth associated with the long‐term approach to equilibrium. The results also suggest methods to use the experimental observations to determine the machine impedance in an qualitative way. Details of the measurements and a development of the theoretical concepts will be given. ©1996 American Institute of Physics.