In this paper, we use the Markov Decision Process (MDP) technique to find the optimal code allocation policy in High-Speed Downlink Packet Access (HSDPA) networks. A discrete stochastic dynamic programming model for the HSDPA downlink scheduler is presented. The model then is solved numerically using value iteration. The system performance when using the resulted optimal policy as compared to Round Robin (RR) is studied using simulation. The behaviour of the value function was observed then used to develop a heuristic scheduling policy. The devised heuristic policy performs very close to the optimal policy. It has much less computational complexity which makes it easy to deploy and with only slight reduction in performance compared to the optimal policy.