Shot boundary detection is a fundamental step for the organization of large video data. A general shot boundary detection method is proposed. To improve the performance of the algorithm and reduce the calculation, smooth intervals inside shots are first concatenated from the original video. After that, features, like intensity pixel-wise difference, color histograms in HSV space and edge histograms in X and Y direction, are extracted from the new video sequence and used as the input vectors to the support vector machine (SVM). Consequently, we use the SVM to classify the frames. The outputs of the SVM are divided into four categories, which are respectively abrupt cuts, gradual changes and etc. After the classification, a detection algorithm is applied to the result sequence of the SVM classification to fulfill the shot boundary detection. Experimental results show that the proposed algorithm produces good detection results.