GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions
2024 (English)In: Proceedings - 2024 ACM/IEEE 46th International Conference on Software Engineering: Companion, ICSE-Companion 2024, Association for Computing Machinery (ACM) , 2024, p. 1-5Conference paper, Published paper (Refereed)
Abstract [en]
Bug-fix benchmarks are fundamental in advancing various subfields of software engineering such as automatic program repair (APR) and fault localization (FL). A good benchmark must include recent examples that accurately reflect t echnologies a nd development practices of today. To be executable in the long term, a benchmark must feature test suites that do not degrade overtime due to, for example, dependencies that are no longer available. Existing benchmarks fail in meeting both criteria. For instance, Defects4J, one of the foremost Java benchmarks, last received an update in 2020. Moreover, full-reproducibility has been neglected by the majority of existing benchmarks. In this paper, we present GitBug-Actions: a novel tool for building bug-fix benchmarks with modern and fully-reproducible bug-fixes. GitBug- Actions relies on the most popular CI platform, GitHub Actions, to detect bug-fixes a nd s martly l ocally e xecute t he CI pipeline in a controlled and reproducible environment. To the best of our knowledge, we are the first t o r ely o n G itHub Actions t o collect bug-fixes. To demonstrate our toolchain, we deploy GitBug- Actions to build a proof-of-concept Go bug-fix benchmark containing executable, fully-reproducible bug-fixes from different repositories. A video demonstrating GitBug-Actions is available at: https://youtu.be/aBWwa1sJYBs.
Place, publisher, year, edition, pages
Association for Computing Machinery (ACM) , 2024. p. 1-5
Series
Proceedings - International Conference on Software Engineering, ISSN 0270-5257
Keywords [en]
Bug Benchmark, Bug Database, GitHub Actions, Program Analysis, Reproducibility, Software Bugs, Software Testing
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:kth:diva-347648DOI: 10.1145/3639478.3640023Scopus ID: 2-s2.0-85194898421OAI: oai:DiVA.org:kth-347648DiVA, id: diva2:1869243
Conference
46th International Conference on Software Engineering: Companion, ICSE-Companion 2024, Lisbon, Portugal, Apr 14 2024 - Apr 20 2024
Funder
Wallenberg AI, Autonomous Systems and Software Program (WASP)
Note
QC 20240613
Part of ISBN 979-840070502-1
2024-06-122024-06-122024-06-25Bibliographically approved