For the d–dimensional reflecting stochastic differential equations (1) with non-smooth boundary and unbounded domain the existence of a strong solution, (weak solution) is obtained under the conditions that the coefficients are less than linear growth and they are non-Lipschitz, (and the diffusion coefficient is non-degenerate, the drift coefficient is bounded and measurable only). Moreover, the Girsanov theorem and the martingale representation theorem with respect to system (1) are also derived. Then by using the Ekeland lemma and the martingale method the existence, necessary and sufficient conditions for an optimal control and an optimal control are obtained. The results are then applied to solve an optimal control problem for a stochastic population model