This paper presents a spoken document summarization scheme using acoustic, prosodic, and semantic information. First, speech recognition confidence is estimated to choose reliable words from the speech transcription. Prosodic information, including pitch and energy, is used for stressed word selection. Latent semantic indexing (LSI) is adopted to identify significant words. Finally, word trigram and semantic dependency is measured to include the syntactic and semantic information for speech summarization. The dynamic programming (DP) algorithm is used to find the best summarization result according to the summarization score estimated from the above five measures. Finally, the summarized result is presented by the concatenation of the summarized speech words. Experimental results indicate that the proposed approach effectively extracts important words and gives a promising speech summary.