We present a novel approach for tracking locally planar regions in an image sequence and their grouping into larger planar surfaces. The tracker recovers the affine transformation of the region and therefore yields reliable point correspondences between frames. Both edges and texture information are exploited in an integrated way, while not requiring the complete region?s contour. The tracker withstands zoom, out-of-plane rotations, discontinuous motion and changes in illumination conditions while achieving real-time performance for a region. Multiple tracked regions are grouped into disjoint coplanarity classes. We first define a coplanarity score between each pair of regions, based on motion and texture cues. The scores are then analyzed by a clique-partitioning algorithm yielding the coplanarity classes that best fit the data. The method works in the presence of perspective distortions, discontinuous planar surfaces and considerable amounts of measurement noise.