The GMDH (group method of data handling) network is an adaptive learning machine based on the principle of heuristic self-organization. In this paper, we apply the GMDH networks to predict software reliability in testing phase. Three kinds of networks: the basic GMDH and its improved versions based on PSS (prediction sum of squared) and AIC (Akaike information criterion), are introduced fro the prediction of the failure-occurrence times observed in testing phase of the software system. In numerical examples, the GMDH networks, the usual MLP (multi-layer perceptron) neural network and existing SRGMs (software reliability growth models) are compared from the viewpoint of predictive performance. It is shown that the GMDH networks can overcome the problem of determining a suitable network size in the use of an MLP neural network, and can provide a more accurate measure in the software reliability assessment than other prediction devices. Further, the problem to determine the optimal software release schedule, which minimizes the relevant expected total software cost, is considered in the framework of the GMDH network architecture.