This paper improves scalability and robustness of our earlier developed approach based on adaptive algorithm for packet routing using reinforcement learning called N Q-routing Optimal Shortest Paths (NQOSP). In contrast with other algorithms that are also based on Reinforcement Learning (RL) methods, the N Q-Routing Optimal Shortest Paths is based on a multi-paths routing technique combined with the QRouting algorithm. In this case, the exploration space is limited to N-best non loop paths in term of hops number (number of routers in a path) leading to a substantial reduction of convergence time. Moreover, each router uses an on line learning module to optimize the path in terms of average packet delivery time. In this paper, we focus on improving the scalability and robustness of our earlier developed approach. The performance of NQOSP is evaluated experimentally with OPNET simulator for different levels of traffic?s load and compared to standard shortest path, N-best algorithm and Q-routing algorithms on large interconnected network. Our Approach prove superior to a classical algorithms and are able to route efficiently even when critical aspects, such as the link broken network, are allowed to vary dynamically.