Change search
ReferencesLink to record
Permanent link

Direct link
Estimation and Comparison of CAD System Performance in Clinical Settings
KTH, School of Engineering Sciences (SCI), Physics.
2005 (English)In: Academic Radiology, ISSN 1076-6332, E-ISSN 1878-4046, Vol. 12, no 6, 687-694 p.Article in journal (Refereed) Published
Abstract [en]

Rationale and Objectives. Computer-aided detection (CAD) systems are frequently compared using free-response receiver operating characteristic (FROC) curves. While there are ample statistical methods for comparing FROC curves, when one is interested in comparing the outcomes of 2 CAD systems applied in a typical clinical setting, there is the additional matter of correctly determining the system operating point. This article shows how the effect of the sampling error on determining the correct CAD operating point can be captured. By incorporating this uncertainty, a method is presented that allows estimation of the probability with which a particular CAD system performs better than another on unseen data in a clinical setting.

Materials and Methods. The distribution of possible clinical outcomes from 2 artificial CAD systems with different FROC curves is examined. The sampling error is captured by the distribution of possible system thresholds of the classifying machine that yields a specified sensitivity. After introducing a measure of superiority, the probability of one system being superior to the other can be determined.

Results. It is shown that for 2 typical mammography CAD systems, each trained on independent representative datasets of 100 cases, the FROC curves must be separated by 0.20 false positives per image in order to conclude that there is a 90% probability that one is better than the other in a clinical setting. Also, there is no apparent gain in increasing the size of the training set beyond 100 cases.

Discussion. CAD systems for mammography are modeled for illustrative purposes, but the method presented is applicable to any computer-aided detection system evaluated with FROC curves. The presented method is designed to construct confidence intervals around possible clinical outcomes and to assess the importance of training set size and separation between FROC curves of systems trained on different datasets.

Place, publisher, year, edition, pages
2005. Vol. 12, no 6, 687-694 p.
Keyword [en]
CAD; performance evaluation; sampling error; confidence interval; mammography; operating point estimation
National Category
Natural Sciences
URN: urn:nbn:se:kth:diva-5394DOI: 10.1016/j.acra.2005.02.005ISI: 000229717100004ScopusID: 2-s2.0-20344405978OAI: diva2:9750
QC 20100819Available from: 2006-03-03 Created: 2006-03-03 Last updated: 2011-11-08Bibliographically approved
In thesis
1. Computer-aided detection and novel mammography imaging techniques
Open this publication in new window or tab >>Computer-aided detection and novel mammography imaging techniques
2006 (English)Doctoral thesis, comprehensive summary (Other scientific)
Abstract [en]

This thesis presents techniques constructed to aid the radiologists in detecting breast cancer, the second largest cause of cancer deaths for western women. In the first part of the thesis, a computer-aided detection (CAD) system constructed for the detection of stellate lesions is presented. Different segmentation methods and an attempt to incorporate contra-lateral information are evaluated.

In the second part, a new method for evaluating such CAD systems is presented based on constructing credible regions for the number of false positive marks per image at a certain desired target sensitivity. This method shows that the resulting regions are rather wide and this explains some of the difficulties encountered by other researchers when trying to compare CAD algorithms on different data sets. In this part an attempt to model the clinical use of CAD as a second look is also made and it shows that applying CAD in sequence to the radiologist in a routine manner, without duly altering the decision criterion of the radiologist, might very well result in suboptimal operating points.

Finally, in the third part two dual-energy imaging methods optimized for contrast-enhanced imaging of breast tumors are presented. The first is based on applying an electronic threshold to a photon-counting digital detector to discriminate between high- and low-energy photons. This allows simultaneous acquisition of the high- and low-energy images. The second method is based on the geometry of a scanned multi-slit system and also allows single-shot contrast-enhanced dual-energy mammography by filtering the x-ray beam that reaches different detector lines differently.

Place, publisher, year, edition, pages
Stockholm: KTH, 2006
Trita-FYS, ISSN 0280-316X ; 2006:06
computer-aided detection, FROC, mammography, optimal operating points, single-shot dual-energy imaging
National Category
Radiology, Nuclear Medicine and Medical Imaging
urn:nbn:se:kth:diva-3861 (URN)91-7178-274-5 (ISBN)
Public defence
2006-03-10, Sal F3, Lindstedtsvägen 26, Stockholm, 10:00
QC 20100819Available from: 2006-03-03 Created: 2006-03-03 Last updated: 2010-08-19Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Bornefalk, Hans
By organisation
In the same journal
Academic Radiology
Natural Sciences

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Altmetric score

Total: 44 hits
ReferencesLink to record
Permanent link

Direct link