This paper presents a system that detects humans climbing fences. After extracting a binary blob contour, the system models the human with an extended star-skeleton representation consisting of the highest contour point and the blob centroid as the two stars. Distances between stars and contour points are computed and smoothed to detect local maximum points. The system then finds certain predicates to form a feature vector for each frame. To analyze the resulting time series, a block based discrete Hidden Markov Model (HMM) is built with predefined action classes {walk, climb up, cross over, drop down} as the state blocks. Each block contains a subset of hidden states and is trained independently to improve the model estimation accuracy with a limited number of sequences. The detection is achieved by decoding the state sequence of the block based HMM. The experiments on image sequences of human climbing fences yield excellent results.