We describe a framework for automated complex human behavior recognition, illustrating important concepts with specific examples drawn from our work on a unique platform designed to understand water crises related behaviors in a public swimming pool. We argue for a hierarchical representation, leveraging on quantitative descriptors to model a behaviour's intermediate semantics. Complex behaviour inference is then demonstrated using a novel regression-based approach based on a modified version of a functional link network which learns quickly and classifies accurately in comparison with other competing decision making schemes.