A smart card based face verification system is proposed in which the feature extraction and decision making is performed on the card. Such an architecture has many privacy and security benefits. As smart cards are limited computational platforms, the face verification algorithms have to be adapted to limit the facial image representations. This minimises the information needed to be sent to the card and lessens the computational load of the template matching. Studies performed on the BANCA and XM2VTS databases demonstrate that by limiting these representations the verification performance of the system is not degraded and that the proposed architecture is a viable one.