We present case studies which show how the paradigm of learning-based testing (LBT) can be successfully applied to black-box requirements testing of industrial reactive systems. For this, we apply a new testing tool LBTest, which combines algorithms for incremental black-box learning of Kripke structures with model checking technology. We show how test requirements can be modeled in propositional linear temporal logic extended by finite data types.We then provide benchmark performance results for LBTest applied to three industrial case studies.
We present case studies which show how the paradigm of learning-based testing (LBT) can be successfully applied to black-box requirements testing of reactive systems. For this we apply a new testing tool LBTest, which combines algorithms for incremental black-box learning of Kripke structures with model checking technology. We show how test requirements can be modeled in propositional linear temporal logic extended by finite abstract data types. We provide benchmark performance results for LBTest applied to two industrial case studies. Finally we present a first coverage study for the tool.
We present an overview of the paradigm of learning-based testing (LBT) for software systems. LBT is a fully automated method for specification-based black-box testing using computational learning principles. It applies the principle of tests as queries, where queries are either generated by a learning algorithm or by a model checker through use of a formal requirements specification. LBT can be applied to automate black-box testing of a variety of different software architectures including procedural and reactive systems. We describe some different testing platforms which have been designed using this paradigm and some representative evaluation results. We also compare LBT with related testing methods.
Learning-based testing (LBT) is an emerging methodology to automate iterative black-box requirements testing of software systems. The methodology involves combining model inference with model checking techniques. However, a variety of optimisations on model inference are necessary in order to achieve scalable testing for large systems.
In this paper we describe the IKL learning algorithm which is an active incremental learning algorithm for deterministic Kripke structures. We formally prove the correctness of IKL. We discuss the optimisations it incorporates to achieve scalability of testing. We also evaluate a black box heuristic for test termination based on convergence of IKL learning.
We present a new algorithm IDS for incremental learning of deterministic finite automata (DFA). This algorithm is based on the concept of distinguishing sequences introduced in (Angluin, 1981). We give a rigorous proof that two versions of this learning algorithm correctly learn in the limit. Finally we present an empirical performance analysis that compares these two algorithms, focussing on learning times and different types of learning queries. We conclude that IDS is an efficient algorithm for software engineering applications of automata learning, such as testing and model inference.
We present a new algorithm IDSfor incremental learning of deterministic finite automata (DFA). This algorithm is based on the concept of distinguishing sequences introduced in [Angluin 1981]. We give a rigorous proof that two versions of this learning algorithm correctly learn in the limit. Finally we present an empirical performance analysis that compares these two algorithms, focussing on learning times and different types of learning queries. We conclude that IDSis an efficient algorithm for software engineering applications of automata learning, such as testing and model inference.
We show how the paradigm of learning-based testing (LBT)can be applied to automate specification-based black-box testing of reactivesystems. Since reactive systems can be modeled as Kripke structures,we introduce an efficient incremental learning algorithm IKL forsuch structures. We show how an implementation of this algorithm combinedwith an efficient model checker such as NuSMV yields an effectivelearning-based testing architecture for automated test case generation(ATCG), execution and evaluation, starting from temporal logic requirements.
We give an introduction to the LBTest tool which implements learning-based testing for reactive systems. It makes use of incremental learning and model checking algorithms to automate: i) test case generation, ii) test execution and iii) test verdict construction. The paper illustrates the tool by means of a pedagogical case study, to enable the user to setup and learn the tool quickly. We provide a usability exercise to support tool evaluation.
In this thesis we investigate the feasibility of learning-based testing (LBT) as a viable testing methodology for reactive systems. In LBT, a large number of test cases are automatically generated from black-box requirements for the system under test (SUT) by combining an incremental learning algorithm with a model checking algorithm. The integration of the SUT with these algorithms in a feedback loop optimizes test generation using the results from previous outcomes. The verdict for each test case is also created automatically in LBT.
To realize LBT practically, existing algorithms in the literature both for complete and incremental learning of finite automata were studied. However, limitations in these algorithms led us to design, verify and implement new incremental learning algorithms for DFA and Kripke structures. On the basis of these algorithms we implemented an LBT architecture in a practical tool called LBTest which was evaluated on pedagogical and industrial case studies.
The results obtained from both types of case studies show that LBT is an effective methodology which discovers errors in reactive SUTs quickly and can be scaled to test industrial applications. We believe that this technology is easily transferrable to industrial users because of its high degree of automation.