This Article 
 Bibliographic References 
 Add to: 
A Formal Methodology for Network Protocol Fingerprinting
November 2011 (vol. 22 no. 11)
pp. 1813-1825
Guoqiang Shu, The Ohio State University, Columbus
David Lee, The Ohio State University, Columbus
Network protocol fingerprinting refers to the process of identifying a protocol implementation by their input and output behaviors. It has been regarded as both a potential threat to network security and also as a useful mechanism for network management. Existing protocol fingerprinting tools share common disadvantages such as being protocol-specific and difficult to automate. This paper proposes a formal methodology for fingerprinting experiments using which we can model a broad spectrum of fingerprinting problems and design-efficient algorithms. We present a formal behavioral model that specifies a protocol principal by its states and transitions, then identify a complete taxonomy of fingerprint matching and discovery problems is identified based on 1) whether the fingerprinting experiment is active or passive and 2) the information available about the specifications and implementations. Algorithms to solve the problems are discussed. In particular, for fingerprint matching algorithm, we propose an efficient PEFSM online separation algorithm for active experiment and concurrent passive testing for passive experiments. For fingerprint discovery problem, there are two cases: if the protocol specification is available as a nondeterministic PEFSM, we apply across verification and back-tracing technique for active and passive discovery, respectively; if no specification is available, we take the machine learning approach and discover the fingerprint by active testing.

[1] Amap Project, http://thc.orgthc-amap/, 2011.
[2] D. Angluin, "Computational Learning Theory: Survey and Selected Bibliography," Proc. 24th Ann. ACM Symp. Theory of Computing (STOC), pp. 351-369, 1992.
[3] O. Arkin and F. Yarochkin, "Xprobe2—A 'fuzzy' Approach to Remote Active Operating System Fingerprinting," http:/www., 2002.
[4] R. Beverly, "A Robust Classifier for Passive TCP/IP Fingerprinting," Proc. Fifth Int'l Workshop Passive and Active Network Measurement (PAM), 2004.
[5] D. Comer and J.C. Lin, "Probing TCP Implementations," Proc. USENIX Summer, pp. 245-255, 1994.
[6] K. Fall and S. Floyd, "Simulation-Based Comparisons of Tahoe, Reno and SACK TCP," Computer Comm. Review, vol. 26, no. 3, pp. 5-21, July 1996.
[7] S. Jaiswal, G. Iannaccone, C. Diot, J. Kurose, and D. Towsley, "Inferring TCP Connection Characteristics through Passive Measurements," Proc. IEEE INFOCOM, 2004.
[8] D. Lee, D. Chen, R. Hao, R.E. Miller, J. Wu, and X. Yin, "A Formal Approach for Passive Testing of Protocol Data Portions," Proc. 10th IEEE Int'l Conf. Network Protocols (ICNP '02), pp. 122-131, 2002.
[9] D. Lee and M. Yannakakis, "Testing Finite-State Machines: State Identification and Verification," IEEE Trans. Computers, vol. 43, no. 3, pp. 306-320, Mar. 1994.
[10] D. Lee and K. Sabnani, "Reverse Engineering of Communication Protocols," Proc. IEEE Int'l Conf. Network Protocols (ICNP), pp. 208-216, Oct. 1993.
[11] D. Lee and M. Yannakakis, "Online Minimization of Transition Systems (Extended Abstract)," Proc. 24th Ann. ACM Symp. Theory of Computing (STOC '92), pp. 264-274, 1992.
[12] D. Lee and M. Yannakakis, "Principles and Methods of Testing Finite State Machines—A Survey," Proc. IEEE, vol. 84, no. 8 pp. 1090-1123, Aug. 1996.
[13] J. Levine, J. Grizzard, and H. Owen, "Using Honeynets to Protect Large Enterprise Networks," Proc. IEEE Security & Privacy, Nov. 2004.
[14] R. Miller, D. Chen, D. Lee, and R. Hao, "Coping with Nondeterminism in Network Protocol Testing," Proc. 17th Int'l Conf. Testing of Comm. Systems (TestCom), 2005.
[15] J. Padhye and S. Floyd, "On Inferring TCP Behavior," Proc. ACM SIGCOMM, pp. 287-298, 2001.
[16] V. Paxson, "Automated Packet Trace Analysis of TCP Implementations," Proc. ACM SIGCOMM, pp. 167-179, 1997.
[17] G. Roua and J. Saffroy, "IP Personality," http:/ippersonality., 2001.
[18] S. Shah, "An Introduction to HTTP Fingerprinting," http:// paper.html , 2004.
[19] G. Shu and D. Lee, "Defending against Internet Host Fingerprinting—Toward an Outermost Barrier of Cyberspace Security," Proc. Working Together: Research & Development (R&D) Partnerships in Homeland Security Conf., 2005.
[20] S. Singh, C. Estan, G. Varghese, and S. Savage, "Automated Worm Fingerprinting," Proc. Symp. Operating Systems Design and Implementation (OSDI), pp. 45-60, 2004.
[21] D. Watson, M. Smart, G. Robert Malan, and F. Jahanian, "Protocol Scrubbing: Network Security through Transparent Flow Modification," IEEE/ACM Trans. Networking, vol. 12, no. 2, pp. 261-273, Apr. 2004.
[22] F. Yarochkin, "Remote OS Detection via TCP/IP Stack Fingerprinting," http:/, 1998.
[23] G. Shu and D. Lee, "Network Protocol System Fingerprinting—A Formal Approach," Proc. IEEE INFOCOM, 2006.
[24] G. Shu and D. Lee, "Message Confidentiality Testing of Security Protocols—Passive Monitoring and Active Checking," Proc. 18th IFIP Int'l Conf. Testing of Comm. Systems (TestCom), 2006.
[25] G. Shu and D. Lee, "Testing Security Properties of Protocol Implementations—A Machine Learning Based Approach," Proc. IEEE Int'l Conf. Distributed Computing Systems (ICDCS), 2007.
[26] G. Shu and D. Lee, "Minutiae: A Formal Methodology for Accurate Protocol Fingerprinting," Proc. Third IEEE Ann. Workshop Secure Network Protocols (NPSEC), 2007.
[27] G. Shu, "Formal Methods and Tools for Testing Network Protocol System Security," PhD thesis, The Ohio State Univ., 2008.
[28] Y. Hsu, G. Shu, and D. Lee, "A Model-Based Approach to Security Flaw Detection of Network Protocol Implementation," Proc. IEEE Int'l Conf. Network Protocols (ICNP), 2008.
[29] G. Shu, Y. Hsu, and D. Lee, "Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning," Proc. 28th IFIP WG6.1 Int'l Conf. Formal Techniques for Networked and Distributed Systems (FORTE), 2008.
[30] D. Angulin, "Learning Regular Sets from Queries and Counterexamples," Information and Computation, vol. 75, pp. 87-106, 1987.
[31] J. Caballero, S. Venkataraman, P. Poosankam, M. Kang, D. Song, and A. Blum, "FiG: Automatic Fingerprint Generation," Proc. 14th Ann. Network and Distributed System Security Symp. (NDSS), Feb. 2007.
[32] D. Brumley, J. Caballero, Z. Liang, J. Newsome, and D. Song, "Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation," Proc. USENIX Security Symp., Aug. 2007.
[33] M. Egele, C. Kruegel, E. Kirda, H. Yin, and D. Song, "Dynamic Spyware Analysis," Proc. USENIX Ann. Technical Conf., June 2007.
[34] A. Petrenko and N. Yevtushenko, "Testing from Partial Deterministic FSM Specifications," IEEE Trans. Computers, vol. 54, no. 9, pp. 1154-1165, Sept. 2005.

Index Terms:
Formal methods, network level security and protection, network management, protocol verification.
Guoqiang Shu, David Lee, "A Formal Methodology for Network Protocol Fingerprinting," IEEE Transactions on Parallel and Distributed Systems, vol. 22, no. 11, pp. 1813-1825, Nov. 2011, doi:10.1109/TPDS.2011.26
Usage of this product signifies your acceptance of the Terms of Use.