Leveraging metamorphic testing to automatically detect inconsistencies in code generator families
2020 (English)In: Software testing, verification & reliability, ISSN 0960-0833, E-ISSN 1099-1689, Vol. 30, no 1, article id e1721Article in journal (Refereed) Published
Abstract [en]
Generative software development has paved the way for the creation of multiple code generators that serve as a basis for automatically generating code to different software and hardware platforms. In this context, the software quality becomes highly correlated to the quality of code generators used during software development. Eventual failures may result in a loss of confidence for the developers, who will unlikely continue to use these generators. It is then crucial to verify the correct behaviour of code generators in order to preserve software quality and reliability. In this paper, we leverage the metamorphic testing approach to automatically detect inconsistencies in code generators via so-called “metamorphic relations”. We define the metamorphic relation (i.e., test oracle) as a comparison between the variations of performance and resource usage of test suites running on different versions of generated code. We rely on statistical methods to find the threshold value from which an unexpected variation is detected. We evaluate our approach by testing a family of code generators with respect to resource usage and performance metrics for five different target software platforms. The experimental results show that our approach is able to detect, among 95 executed test suites, 11 performance and 15 memory usage inconsistencies.
Place, publisher, year, edition, pages
John Wiley and Sons Ltd , 2020. Vol. 30, no 1, article id e1721
Keywords [en]
code generators, metamorphic testing, non-functional properties, software quality, test automation, test oracle, Computer software selection and evaluation, Software design, Software reliability, Testing, Non functional properties, Test oracles, Software testing
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:kth:diva-272306DOI: 10.1002/stvr.1721ISI: 000503568200001Scopus ID: 2-s2.0-85076878749OAI: oai:DiVA.org:kth-272306DiVA, id: diva2:1426612
Note
QC 20200427
2020-04-272020-04-272022-06-26Bibliographically approved