In the recognition of Voice Over Internet Protocol(VoIP) speech, packet losses pose a challenge that is generally addressed by packet loss concealment (PLC)techniques. But improper concealment by these PLC techniques result in unreliable observations that contribute adversely to the Viterbi decoding step and result in misrecognitions. We propose a graphical model based decoding architecture that can skip unreliable observations and hence the corresponding states, so that recognition accuracy improves in spite of improper concealment. Experimental validation of the proposed skip decoder is carried out using the loss-concealed speech samples of isolated words. Results indicate the efficacy of the skip decoder in the improvement of the recognition accuracy for different lengths of burst losses and different PLC schemes.