There have been various studies on rule extraction from data such as ID3 in machine learning. Recently rule extraction using neural networks is attracting wide attention because of its simplicity and flexibility. This is, however, very hard due mainly to distributed representations on hidden layers. A basic idea of rule extraction proposed here is the elimination of unnecessary connections by a structural learning with forgetting (SLF). The proposed rule extraction is based solely on data, i.e., without initial theories and pre-processing. To evaluate its effectiveness, SLF as well as BP learning and ID3 are applied to a classification of mushrooms, a MONKS problem and a promotor recognition in DNA sequences.