A Markovian reinforcement-learning task can be dealt with by learning a direct mapping from states to actions or values, or from state-action pairs to values. However, this may involve a difficult pattern recognition problem when the state space is large. This paper shows that using internal state, called “supportive state”, may alleviate this problem-presenting an argument against the tendency to almost automatically use a direct mapping when the task is Markovian. This point is demonstrated in simulation experiments of an agent controlled by a neural network capable of learning the strategy of direct mapping as well as internal state, combining Q(\math)-learning and recurrent neural networks in a new way. The trade-off between the two strategies is investigated in more detail, focusing in particular on border cases.