Many deployed systems for human motion tracking and detection are found inadequate when applied on hostile outdoor environments. This paper provides insights into this problem by developing an outdoor aquatic surveillance system, which detects swimmers within the hostile environment of an outdoor public swimming pool. A novel block-based background model and thresholding-with-hysteresis methodology is proposed to extract swimmers amid reflections, ripples, splashes and lighting changes. The problem of partial occlusion between swimmers is resolved based on a proposed Markov Random Field framework. The algorithm has been incorporated into a live system with robust results for different challenging outdoor pool conditions.