In nonlinear optimal control problems, open-loop solutions from a fixed initial condition are much easier to compute than closed-loop solutions which do not depend on initial conditions. Two methods of using neural networks to approximate the optimal feedback controller are discussed. The indirect method uses a neural network to interpolate the whole field of extremals obtained from open-loop calculation. The direct method directly trains a neural network such that a general nonlinear optimal control performance index is minimized. The novelty of the modified backpropagation training is the requirement of the jacobian matrix of the neural network function. Simulation studies show that the closed-loop solution can be made to be arbitrarily close to the optimal open-loop solution with initial conditions chosen from a nontrivial subset of the state space.