This paper describes a novel approach for detecting unattended packages in surveillance video. Unlike the traditional approach to just detecting stationary objects in monitored scenes, our approach detects unattended packages based on accumulated knowledge about human and non-human objects from continuous object tracking and classification. We design different reasoning rules for detecting different scenarios of the unattended package events. In the case where a package is left unattended by a single person explicitly, a rule using human activity recognition is introduced to decide the package ownership. In the case where a suspicious package is dropped down by a group of humans or under heavy occlusions, a rule based on historic tracking and classification information is proposed. Furthermore, an additional rule is given to reduce false alarms that may happen with traditional stationary object detection methods.