This paper proposes a novel speech processing framework, where all of the speech data are once encoded into universal phonetic code (UPC) sequences and speech processing systems, such as speech recognition, retrieval, digesting, are constructed on this UPC domain. First of all, we introduce an IPA-based sub-phonetic segment (SPS) set as the UPC to deal with multilingual speech. In the UPC (SPS) domain, each UPC accompanies a reference acoustic model which is independent of real acoustic models used in the encoding process. Processing, such as recognition, in the UPC domain is conducted based on the distance between UPC sequences estimated by using the reference acoustic models. We confirm the proposed framework by constructing a speech recognition and a vocabulary-free speech retrieval system on the SPS domain. We show several experimental results on these systems, using Japanese and English speech data sets.