In this paper, the design and implementation results of a system on a chip (SOC) based speech recognition system are presented. The hidden markov model (HMM) is used for the speech recognition. In order to implement this in SOC, the various tasks required are optimally partitioned between hardware and software. The SOC, housed in Altera UP3 kit, consists of both Altera Nios II soft core processor and custom hardware blocks for computationally intensive blocks such as Viterbi decoder. The preprocessing and training of HMM are implemented in software (using C program). The Viterbi decoding is implemented in hardware for real time recognition. It is also implemented in software for verification and comparison. It is observed that the hardware implementation of viterbi block is 80 times faster than the software approach using C program. The speech recognition system is trained for digits 0-9 uttered by three male speakers. It is tested with trained speakers utterances and three new untrained speakers utterances. An over all recognition accuracy of 94.8% is achieved. Extension of this work for larger vocabulary size is under progress.