Neural networks have received a great deal of attention over the last few years. They are being used in the areas of prediction and classification, areas where regression models and other related statistical techniques have traditionally been used. In this paper we discuss neural networks and compare them to regression models. We start by exploring the history of neural networks. This includes a review of relevant literature on the topic of neural networks. Neural network nomenclature is then introduced, and the backpropagation algorithm, the most widely used learning algorithm, is derived and explained in detail. A comparison between regression analysis and neural networks in terms of notation and implementation is conducted to aid the reader in understanding neural networks. We compare the performance of regression analysis with that of neural networks on two simulated examples and one example on a large dataset. We show that neural networks act as a type of nonparametric regression model, enabling us to model complex functional forms. We discuss when it is advantageous to use this type of model in place of a parametric regression model, as well as some of the difficulties in implementation.