We propose a new theoretical framework for generalizing the traditional notion of covariance. First, we discuss the role of pairwise cross-cumulants by introducing a cluster expansion technique for the cumulant generating function. Next, we introduce a novel concept of symmetry decomposition of probability density functions according to the C_4v group. By utilizing the irreducible representations, generalized covariances are explicitly defined, and their utility is demonstrated using an analytically solvable model.