Subscribe

Issue No.03 - May-June (2012 vol.38)

pp: 629-641

Robert Mark Hierons , Brunel University, Middlesex

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TSE.2011.45

ABSTRACT

The problem of deciding whether an observed behavior is acceptable is the oracle problem. When testing from a finite state machine (FSM), it is easy to solve the oracle problem and so it has received relatively little attention for FSMs. However, if the system under test has physically distributed interfaces, called ports, then in distributed testing, we observe a local trace at each port and we compare the set of local traces with the set of allowed behaviors (global traces). This paper investigates the oracle problem for deterministic and nondeterministic FSMs and for two alternative definitions of conformance for distributed testing. We show that the oracle problem can be solved in polynomial time for the weaker notion of conformance ({\sqsubseteq_w}) but is NP-hard for the stronger notion of conformance ({\sqsubseteq_s}), even if the FSM is deterministic. However, when testing from a deterministic FSM with controllable input sequences, the oracle problem can be solved in polynomial time and similar results hold for nondeterministic FSMs. Thus, in some cases, the oracle problem can be efficiently solved when using \sqsubseteq_s and where this is not the case, we can use the decision procedure for \sqsubseteq_w as a sound approximation.

INDEX TERMS

Software engineering/software/program verification, software engineering/testing and debugging, systems and software, distributed systems, finite state machine, nondeterminism, test oracle, controllability, local observability.

CITATION

Robert Mark Hierons, "Oracles for Distributed Testing",

*IEEE Transactions on Software Engineering*, vol.38, no. 3, pp. 629-641, May-June 2012, doi:10.1109/TSE.2011.45REFERENCES

- [1] W. Chen and H. Ural, "Synchronizable Test Sequences Based on Multiple UIO Sequences,"
IEEE/ACM Trans. Networking, vol. 3, no. 2, pp. 152-157, Apr. 1995.- [2] R. Dssouli and G. von Bochmann, "Error Detection with Multiple Observers,"
Proc. Int'l Conf. Protocol Specification, Testing and Verification V, pp. 483-494, 1985.- [3] R. Dssouli and G. von Bochmann, "Conformance Testing with Multiple Observers,"
Proc. Int'l Conf. Protocol Specification, Testing and Verification VI, pp. 217-229, 1986.- [4] R.M. Hierons and H. Ural, "The Effect of the Distributed Test Architecture on the Power of Testing,"
The Computer J., vol. 51, no. 4, pp. 497-510, 2008.- [5] A. Khoumsi, "A Temporal Approach for Testing Distributed Systems,"
IEEE Trans. Software Eng., vol. 28, no. 11, pp. 1085-1103, Nov. 2002.- [6] G. Luo, R. Dssouli, and G. von Bochmann, "Generating Synchronizable Test Sequences Based on Finite State Machine with Distributed Ports,"
Proc. Sixth IFIP Workshop Protocol Test Systems, pp. 139-153, 1993.- [7] B. Sarikaya and G. von Bochmann, "Synchronization and Specification Issues in Protocol Testing,"
IEEE Trans. Comm., vol. 32, no. 4, pp. 389-395, Apr. 1984.- [8] K.-C. Tai and Y.-C. Young, "Synchronizable Test Sequences of Finite State Machines,"
Computer Networks and ISDN Systems, vol. 30, no. 12, pp. 1111-1134, 1998.- [9] T.S. Chow, "Testing software Design Modelled by Finite State Machines,"
IEEE Trans. Software Eng., vol. 4, no. 3, pp. 178-187, May 1978.- [10] E. Farchi, A. Hartman, and S. Pinter, "Using a Model-Based Test Generator to Test for Standard Conformance,"
IBM Systems J., vol. 41, no. 1, pp. 89-110, www.research.ibm. com/journal/sj/411farchi.pdf , 2002.- [11] M. Barnett, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veanes, "Towards a Tool Environment for Model-Based Testing with AsmL,"
Proc. Int'l Workshop Formal Approaches to Testing of Software, pp. 252-266, 2003.- [12] W. Grieskamp, Y. Gurevich, W. Schulte, and M. Veanes, "Generating Finite State Machines from Abstract State Machines,"
Proc. ACM SIGSOFT Symp. Software Testing and Analysis, pp. 112-122, 2002.- [13] F.C. Hennie, "Fault-Detecting Experiments for Sequential Circuits,"
Proc. Fifth Ann. Symp. Switching Circuit Theory and Logical Design, pp. 95-110, Nov. 1964.- [14] E.P. Moore, "Gedanken-Experiments,"
Automata Studies, C. Shannon and J. McCarthy, eds., Princeton Univ. Press, 1956.- [15] A. Petrenko, S. Boroday, and R. Groz, "Confirming Configurations in EFSM Testing,"
IEEE Trans. Software Eng., vol. 30, no. 1, pp. 29-42, Jan. 2004.- [16] A. Petrenko and N. Yevtushenko, "Testing from Partial Deterministic FSM Specifications,"
IEEE Trans. Computers, vol. 54, no. 9, pp. 1154-1165, Sept. 2005.- [17] J. Tretmans, "Conformance Testing with Labelled Transitions Systems: Implementation Relations and Test Generation,"
Computer Networks and ISDN Systems, vol. 29, no. 1, pp. 49-79, 1996.- [18] S. Boyd and H. Ural, "The Synchronization Problem in Protocol Testing and Its Complexity,"
Information Processing Letters, vol. 40, no. 3, pp. 131-136, 1991.- [19] D. Lee and M. Yannakakis, "Principles and Methods of Testing Finite-State Machines—A Survey,"
Proc. IEEE, vol. 84, no. 8, pp. 1089-1123, Aug. 1996.- [20] R.M. Hierons, M.G. Merayo, and M. Núñez, "Implementation Relations for the Distributed Test Architecture,"
Proc. 20th IFIP TC 6/WG 6.1 Int'l Conf. Testing of Software and Comm. Systems, pp. 200-215, 2008.- [21] R.M. Hierons, M.G. Merayo, and M. Núñez, "Implementation Relations and Test Generation for Systems with Distributed Interfaces," submitted for publication.
- [22] R.M. Hierons, "Controllable Testing from Nondeterministic Finite State Machines with Multiple Ports,"
IEEE Trans. Computers, vol. 60, no. 12, pp. 1818-1822, Dec. 2011.- [23] J.L. Jacob, "Refinement of Shared Systems,"
Theory and Practice of Refinement: Approaches to the Formal Development of Large-Scale Software Systems, J. McDermid, ed., pp. 27-36, Butterworths, 1989.- [24] C. Kloukinas, G. Spanoudakis, and K. Mahbub, "Estimating Event Lifetimes for Distributed Runtime Verification,"
Proc. 20th Int'l Conf. Software Eng. and Knowledge Eng., pp. 117-122, 2008.- [25] L. Cacciari and O. Rafiq, "Controllability and Observability in Distributed Testing,"
Information and Software Technology, vol. 41, nos. 11/12, pp. 767-780, 1999.- [26] O. Rafiq and L. Cacciari, "Coordination Algorithm for Distributed Testing,"
The J. Supercomputing, vol. 24, no. 2, pp. 203-211, 2003.- [27] A. Bauer, M. Leucker, and C. Schallhart, "Model-Based Runtime Analysis of Distributed Reactive Systems,"
Proc. 17th Australian Software Eng. Conf., pp. 243-252, 2006.- [28] P.S. Dodd and C.V. Ravishankar, "Monitoring and Debugging Distributed Real-Time Programs,"
Software—Practice and Experience, vol. 22, no. 10, pp. 863-877, 1992.- [29] D. Gunter, B. Tierney, B. Crowley, M. Holding, and J. Lee, "Netlogger: A Toolkit for Distributed System Performance Analysis,"
Proc. Eighth Int'l Symp. Modeling, Analysis and Simulation of Computer and Telecomm. Systems, pp. 267-273, 2000.- [30] M. Mansorui-Samani and M. Sloman, "Monitoring Distributed Systems,"
IEEE Network, vol. 7, no. 6, pp. 20-30, Nov. 1993.- [31] M. Zulkernine and R.E. Seviora, "A Compositional Approach to Monitoring Distributed Systems,"
Proc. Int'l Conf. Dependable Systems and Networks, pp. 763-772, 2002.- [32] E. Brinksma, L. Heerink, and J. Tretmans, "Factorized Test Generation for Multi-Input/Output Transition Systems,"
Proc. IFIP TC6 11th Int'l Workshop Testing Communicating Systems, pp. 67-82, 1998.- [33] L. Heerink and J. Tretmans, "Refusal Testing for Classes of Transition Systems with Inputs and Outputs,"
Proc. Int'l Conf. Formal Description Techniques for Distributed Systems and Comm. Protocols and Protocol Specification, Testing and Verification, pp. 23-38, 1997.- [34] Z. Li, J. Wu, and X. Yin, "Testing Multi Input/Output Transition System with All-Observer,"
Proc. 16th IFIP Int'l Conf. Testing of Communicating Systems, pp. 95-111, 2004.- [35] T.J. Schaefer, "The Complexity of Satisfiability Problems,"
Proc. 10th Ann. ACM Symp. Theory of Computing, pp. 216-226, 1978.- [36] S. Haar, C. Jard, and G.-V. Jourdan, "Testing Input/Output Partial Order Automata,"
Proc. Int'l Conf. Testing of Software and Communicating Systems, pp. 171-185, 2007.- [37] G. von Bochmann, S. Haar, C. Jard, and G.-V. Jourdan, "Testing Systems Specified as Partial Order Input/Output Automata,"
Proc. 20th IFIP TC 6/WG 6.1 Int'l Conf. Testing of Software and Communicating Systems, pp. 169-183, 2008. |