Bundle ajustment is used to obtain accurate visual reconstructions by minimizing the reprojection error. The coordinate frame ambiguity, or more generality the gauge freedoms, has been dealt with in different manners. It has often been reported that standard bundle adjustment algorithms were not gauge invariant: two iterations within different gauges can lead to geometrically very different results. Surprisingly, most algorithms do not exploit gauge freedoms to improve performances. We consider this issue. We analyze theoretically the impact of the gauge on standard algorithms. We show that a sufficiently general damping matrix in Levenberg-Marquardt iteration can be used to implicitly reproduce a gauge transformation. We show that if the damping matrix is chosen such that the decrease in the reprojection error is maximized, then the iteration is gauge invariant. Experimental results on simulated and real data show that our gauge invariant bundle adjustment algorithm outperforms existing ones in terms of stability.