From the probability distribution of the phase angle between two vectors perturbed by Gaussian noises, the formulae of bit error rate(BER) for the performance analysis of Gaussian minimum shift keying(GMSK) system applying two-bit differential detection and decision feedback(DF) is derived in this paper. Considering the m-bits dock-tailed sequence, the new formulae of GMSK modulated phase and the time-varying signal to noise ratio (TVSNR) of demodulated signal are presented, and it is proved that the relationship between the TVSNR of demodulated signal and the size of eye opening is not inevitable. Simulation results show that the theoretical investigation gives analogous results with the Monte Carlo error counting(MCEC) technique. The formulae presented are useful for the performance analysis of systems using GMSK as modulating and demodulating method, for instance, the analysis of synchronous performance of Frequency-hopping communication system.