Understanding the mechanisms of reactive sputtering is thought to be crucial to obtain a better process control and widen applications of reactive sputtering. A model simulating reactive gas mass balance changes in reactive sputtering is described. The model is based on physical mechanisms involved in target and wall gettering behavior. This enables calculation of time‐dependent condition changes that occur until the process reaches a steady state, without assuming how the process reaches a steady state. Another important feature of the model is that hysteresis curves are obtained as a result of the calculation of time‐dependent target condition changes. This approach to obtain hysteresis curves is more similar to an actual reactive sputtering process. The proposed model clearly displays changes in target coverage, sputtered flux, and reactive gas partial pressure during compound‐layer formation and sputter etching processes as a function of elapsed time. Effects of reactive gas flow rate, pumping speed, and sputtering current on compound‐layer formation and sputter etching are also discussed. Hysteresis curves are yielded as a result of compound‐layer formation and sputter etching calculations. The obtained curves are in accord with experimental results. Further, the dependence of hysteresis on pumping speed, sputtering current, and the difference of sputtering yields between metallic and compound‐covered targets was investigated, resulting in showing a good agreement to that obtained experimentally. Mechanisms of hysteresis formation are also discussed on the basis of obtained results. It is concluded that hysteresis is formed because of the difference of gettering capacity before and after the target surface condition transition.