kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A comprehensive study of automatic program repair on the QuixBugs benchmark
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Theoretical Computer Science, TCS.ORCID iD: 0000-0003-4807-2110
Université Polytechnique Hauts-de-France, France.
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS.ORCID iD: 0000-0002-1996-6134
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Theoretical Computer Science, TCS.ORCID iD: 0000-0003-3505-3383
2021 (English)In: Journal of Systems and Software, ISSN 0164-1212, E-ISSN 1873-1228, Vol. 171, article id 110825Article in journal (Refereed) Published
Abstract [en]

Automatic program repair papers tend to repeatedly use the same benchmarks. This poses a threat to the external validity of the findings of the program repair research community. In this paper, we perform an empirical study of automatic repair on a benchmark of bugs called QuixBugs, which has been little studied. In this paper, (1) We report on the characteristics of QuixBugs; (2) We study the effectiveness of 10 program repair tools on it; (3) We apply three patch correctness assessment techniques to comprehensively study the presence of overfitting patches in QuixBugs. Our key results are: (1) 16/40 buggy programs in QuixBugs can be repaired with at least a test suite adequate patch; (2) A total of 338 plausible patches are generated on the QuixBugs by the considered tools, and 53.3% of them are overfitting patches according to our manual assessment; (3) The three automated patch correctness assessment techniques, RGTEvosuite, RGTInputSampling and GTInvariants, achieve an accuracy of 98.2%, 80.8% and 58.3% in overfitting detection, respectively. To our knowledge, this is the largest empirical study of automatic repair on QuixBugs, combining both quantitative and qualitative insights. All our empirical results are publicly available on GitHub in order to facilitate future research on automatic program repair. 

Place, publisher, year, edition, pages
Elsevier Inc. , 2021. Vol. 171, article id 110825
Keywords [en]
Automatic program repair, Bug benchmark, Patch correctness assessment, Software engineering, Assessment technique, Automatic programs, Empirical studies, External validities, Overfitting, Repair tools, Research communities, Automatic test pattern generation
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:kth:diva-285291DOI: 10.1016/j.jss.2020.110825ISI: 000592499600001Scopus ID: 2-s2.0-85091196139OAI: oai:DiVA.org:kth-285291DiVA, id: diva2:1506248
Funder
Wallenberg AI, Autonomous Systems and Software Program (WASP)
Note

QC 20201202

Available from: 2020-12-02 Created: 2020-12-02 Last updated: 2024-03-18Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Ye, HeDurieux, ThomasMonperrus, Martin

Search in DiVA

By author/editor
Ye, HeDurieux, ThomasMonperrus, Martin
By organisation
Theoretical Computer Science, TCSSoftware and Computer systems, SCS
In the same journal
Journal of Systems and Software
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 227 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf