Credit risk assessment is a basic and critical factor in credit risk management. In addition to conventional statistical method, neural network, decision tree and Support Vector Machine are the popular methods in this field in recent years. However, they all have weakness in two aspects: poor classification accuracy for unbalanced data and poor interpretability in real applications. A novel method, called Least Squares Support Feature Machine (LS-SFM), is proposed to reduce the misclassification cost and achieve an interpretable model by introducing the single feature kernel and sampling method. One important character of LS-SFM is that it can deliver the significance of each feature to users Our experiment on a real credit card dataset shows good performance. LS-SFM outperforms some well-known methods in several aspects.