Automatically understanding events happening at a site is the ultimate goal of visual surveillance system. This paper investigates the challenges faced by automated surveillance systems operating in hostile conditions and demonstrates the developed algorithms via a system that detects water crises within highly dynamic aquatic environments. An efficient segmentation algorithm based on robust block-based background modeling and thresholding-with-hysteresis methodology enables swimmers to be reliably detected amid reflections, ripples, splashes and rapid lighting changes. Partial occlusions are resolved using a Markov Random Field framework that enhances the tracking capability of the system. Visual indicators of water crises are identified based on professional knowledge of water crises detection, based on which a set of swimmer descriptors has been defined. Through seamlessly fusing the extracted swimmer descriptors based on a novel functional link network, the system achieves promising results for water crises detection. The developed algorithms have been incorporated into a live system with robust performance for different hostile environments faced by an outdoor swimming pool.