In this note we investigate a stochastic dynamic programming problem with countable state space and finite action space. An approximation procedure is given for the numerical solution by a decomposition algorithm of the state space. The optimal policies and minimal value functions of the so-constructed Markovian Decision Process are calculated' by dynamic programming. Convergence theorems are proved and bounds for the minimal value function are obtained.