This paper describes a method of generating a Kanji hierarchical structured dictionary for stroke-number and stroke-order free handwriting recognition based on sub-stroke HMM. In stroke-based methods, a large number of stroke-order variations can be easily expressed by just adding dierent stroke sequences to the dictionary and it is not necessary to train new reference patterns. The hierarchical structured dictionary has an advantage that thousands of stroke-order variations of Kanji characters can be produced using a small number of stroke-order rules defining Kanji parts. Moreover, the recognition speed is fast since common sequences are shared in a substroke network, even if the total number of stroke-order combinations becomes enormous practically. In experiments, 300 dierent stroke-order rules of Kanji parts were statistically chosen by using 60 writers? handwritings of 1,016 educational Kanji characters. By adding these new stroke-order rules to the dictionary, about 9,000 variations of dierent stroke-orders were generated for 2,965 JIS 1st level Kanji characters. As a result, we successfully improved the recognition accuracy from 82.6% to 90.2% for stroke-order free handwritings.