Significance of results in cancer imaging
- Rodney H. Reznek1
© International Cancer Imaging Society 2001
Published: 5 May 2015
The effective management of patients with cancer requires a multidisciplinary team approach with the diagnostic radiologist playing an extremely important role in that team. Increasingly it is realized that it is often the responsibility of the radiologist to understand and elucidate the significance of the findings of a test. Its significance lies not only in the clinical context but also in appreciating the impact that the test will have on the patient’s outcome. The latter requires a knowledge of the cost-effectiveness of the use of any imaging technique.
Evaluation of the effect of imaging
Impact on health
The positive effect of one level is determined by the level above and in turn determines the possibility of a positive result at the level below.
Technical performance relates to the ability to obtain high image quality in a reasonable time frame and whether these images permit correct interpretation.
Diagnostic performance is concerned with the ability of the technique to identify disease correctly. Thus, diagnostic performance is a measure of sensitivity, specificity, positive predictive value, negative predictive value and accuracy of the technique in a given clinical situation. This is a familiar method of evaluating imaging in cancer and the major method by which different imaging techniques are compared. Thus, the decision to use one imaging technique for staging cancer in preference to another is frequently based on information provided on diagnostic performance. While it is not possible to discuss the use of statistics in detail, it is important to recognize that studies should be designed to answer an hypothesis and that the help of a statistician to design a study is likely to yield enormous benefits by reducing inappropriate methodology and bias.
Diagnostic impact is determined by the influence of the result of imaging on the clinician’s diagnostic confidence and by the ability of the new technology to replace older established methods. Displacement of older techniques by new imaging modalities is easy to demonstrate. For example, lymphangiography has now become obsolete in the staging of several cancers and myelography has also been superseded by magnetic resonance imaging (MRI) in the investigation of spinal cord compression[4,5].
Therapeutic impact reflects the alteration in management of a patient based on results of imaging. Dixon et al. recorded changes in the proposed treatment in 182 of 200 patients referred for MRI of the head and spine and, in the same group of patients, surgery was considered to be appropriate in 50 patients before MRI, but in only 28 patients following the results of the examination.
Impact on health is much more difficult, if not impossible, to evaluate, particularly in oncology when diagnostic information may be in advance of the ability to treat the disease. However, progress in research in both diagnosis and treatment of cancer can only be made by furthering our understanding of the natural processes of therapeutic response and tumour regrowth. In this context, therefore, imaging has an important role in cancer even if there is no demonstrable impact on health. Furthermore, it must be emphasized that although imaging itself cannot make an impact on outcome, the results of imaging may directly influence management allowing the clinician to make the optimum therapeutic decision. In this way diagnostic imaging through therapy does make an important contribution to final outcome.
The diagnostic impact of imaging is most frequently made on the basis of studies designed to evaluate the ability of a technique to detect cancer accurately. In a review entitled ‘A guide to clinical epidemiology for radiologists’, Goldin and Sayre commented that the poor understanding by physicians of the principles of statistical analysis weakens many investigations. Their review discusses the different methods of statistical analysis and basic concepts used to select the appropriate technique and to interpret the results, and is recommended as an excellent overview of the subject.
In the text of the chapters that follow, many references are made to sensitivity, specificity, positive predictive value, negative predictive value and accuracy. Advising on the judicious use of imaging studies in the staging and evaluation of malignancy requires a thorough understanding of these basic tests of efficacy and of the receiver-operator characteristics curve. These terms are defined below:
The specificity is also called the false-positive rate of the test.
The sensitivity and specificity of a test are generally independent of disease prevalence and are therefore often called the intrinsic operating characteristics of the test. On the other hand the PPV (and NPV) and accuracy are highly dependent on the prevalence of the disease and cannot be generalized over settings where the prevalence varies. For this reason, reports of sensitivity and specificity are more reliable than tests of PPV and accuracy, which are greatly influenced by regional variation of disease prevalence.
Thus, as the criteria for calling a test result positive are made more stringent, specificity improves at the expense of sensitivity. Conversely, as the criteria are relaxed, sensitivity improves while specificity diminishes.
The fundamental principle illustrated by the ROC curve is that there is an inherent limit to the diagnostic efficacy of a test. Once this limit has been reached, the interpreter can only improve sensitivity at the expense of specificity and vice versa. The ROC curve can be used to select the ‘best’ cut-off criteria for positivity taking the positive predictive value and the relative costs (in terms of patient outcome) of false-positive and false-negative rest results into account. This has particular relevance in the use of imaging in staging cancer where cut-off criteria for positive results are constantly being decided. An example of this is on deciding on the upper limit of normal size for lymph nodes on cross-sectional imaging.
An understanding of the ROC curve is therefore essential for all radiologists and oncologists interpreting the results of imaging in staging cancer. First, the curve displays explicitly the trade-off between sensitivity and specificity which results from varying the criterion for interpretation. Second, it provides a graphical summary of how well a test performs for each method of interpretation, allowing one to compare two or more tests without the necessity of having to stipulate the positive criterion for each test.
Kappa statistics are used to demonstrate the agreement between observers or different tests.
Interobserver agreement (Kappa)
Altman describes well how to measure interobserver agreement, using as data the assessments of 85 xeromammograms by two radiologists (A and B) where the xeromammogram reports are given as one of four results: normal, benign disease, suspected cancer, cancer.
As Altman points out, the simplest approach is to count how many exact agreements were observed between A and B, which from Table 1 is 54/85 = 0.64. However, the disadvantages with this method of merely quoting a 64% measure of agreement is that it does not take into account where the agreements occurred and also the fact that one would expect a cetain amount of agreement between radiologist A and radiologist B purely by chance, even if they were guessing their assessments.
The complete theory underpinning the kappa (κ) test, including the calculation of confidence intervals and including a weighted kappa test where all disagreements are not treated equally, has been given by Altman.
Calculation of the expected frequencies for the kappa test, after Altman
33 × (28/85) = 10.87
22 × (38/85) = 9.84
29 × (16/85) = 5.46
1 × (3/85) = 0.04
The radiologist should undoubtedly be at the forefront of deciding which test should be used in evaluating patients with malignant disease and the appropriate and judicious use of radiological technology is a formidable challenge.
Based on the discussion above, it is clear that the proper use of imaging in cancer is a complex issue and at best only guidelines on the appropriate use of imaging techniques can be provided in the chapters which follow. Nevertheless, there are certain important issues that need to be addressed in the choice of a particular imaging technique which relate not only to technical and diagnostic performance but also to the purpose of imaging in an individual patient.
Imaging may be requested to answer a specific clinical question in an individual patient on cancer therapy or it may be requested as a routine investigation at the time of presentation for diagnostic and staging purposes. In those tumours where established therapy is available imaging is required to measure therapeutic response. Imaging also has a major role in supporting clinical trials of new therapeutic agents and in this situation is used more frequently during the course of cancer than when used as a tool for management decisions. Imaging to support clinical trials is an increasingly important role for the radiologist with an interest in oncology. The very high accuracy of and reproducibility of cross-sectional imaging (particularly computed tomography (CT) and MRI) makes it extremely well suited to Phase II trials in which the oncologist is assessing the biological activity of new treatments. In Phase III trials, comparing the results of different treatments, survival is usually the final arbiter. If the size of the patient group is large enough, sophisticated staging is unnecessary as the stage will be randomized out. In practice, however, the groups tend to be small and one of the prognostic variables, namely the varying stage of the disease, can be removed from the study by achieving more accurate staging through imaging. Furthermore, in patients with advanced disease, where there is no obvious difference in survival and the end-point of the study becomes the response rate rather than survival, accurate imaging becomes an extremely valuable research tool.
An important impact of the use of sophisticated techniques to stage patients with cancer is the apparent continuous improvement in cancer survival rates reported over the last 25 years. Although this is quickly and easily attributable to earlier diagnosis and new and more effective treatments, the effect of more accurate staging may to some extent explain these improved results[11–13]. Feinstein et al. found that a 1977 cohort of patients who had undergone lung cancer treatment survived significantly longer in each of three TNM subcategories than a cohort managed in the 1950s and 1960s; a finding which is not surprising. When, however, he staged the recent cohort on clinical grounds only, without the benefit of ultrasonography, CT and nuclear medicine, these survival differences disappeared. It was apparent that the improved survival rates were mainly an artefact of better staging; patients in the lower stages with clinically occult (usually nodal) disease were being identified with better imaging and were being placed in a more advanced stage (‘stage migration’). Better staging led to benefit to all; in the lower stages, patients with occult metastases would be removed with benefit to those stages; in the higher stages, those patients with a lower tumour burden would be added to those with a higher one, with improvement in survival rates. Thus while individual prognosis did not change overall, survival in each stage improved. The stage migration phenomenon occurs when comparisons are made between groups of patients who have undergone less or more thorough staging techniques and as such is likely to occur when the comparisons are made over a time period which spans the introduction of new technology. It has been noted with numerous tumours including metastatic germ cell tumours[6,12] and gastric cancers.
Imaging may be used for surveillance of patients with no clinical evidence or imaging evidence of disease in order to identify relapse as early as possible. In patients with clinical suspicion of relapse, again imaging is required to detect recurrence in the previously treated patient. The choice of an imaging technique in this clinical setting depends on the ability of the different imaging methods, not only to identify an abnormality, but also to characterize a lesion and distinguish benign from malignant pathology in the presence of previously treated normal tissues which may have been damaged by therapy.
In all the situations outlined above, the imaging modality chosen will depend upon local factors which include the availability of equipment, the expertise of medical and ancillary personnel and the demands made on imaging by the workload of the department.
The choice of a technique for different tumour types For a given imaging technique examination protocols should be agreed for every tumour
The timing of imaging in relation to treatment should be agreed
Follow-up studies should also be performed to an agreed protocol
Finally, the impact of diagnostic imaging in cancer is enormously improved by working in a multidisciplinary team with regular clinicoradiological review of imaging studies in relation to management decisions.
- MacKenzie R, Dixon AK. Review: measuring the effects of imaging: an evaluation framework. Clin Radiol 1995; 50: 513–8.View ArticlePubMedGoogle Scholar
- Donabedian A. Evaluating the quality of medical care. Millbank Memorial Fund Quarterly 1966; 44: 166–206.View ArticleGoogle Scholar
- Fineberg HV, Wittenberg J, Ferrucci JT. The clinical value of body computed tomography over time and technologic change. Am J Roentgenol 1983; 141: 1067–72.View ArticleGoogle Scholar
- Libson E, Polliack A, Bloom RA. Value of lymphangiography in the staging of Hodgkin lymphoma. Radiology 1994; 193: 757–9.View ArticlePubMedGoogle Scholar
- Williams MP, Cherryman GR, Husband JES. Magnetic resonance imaging in suspected metastatic spinal cord compression. Clin Radiol 1989; 40: 286–90.View ArticlePubMedGoogle Scholar
- Dixon AK, Southern JP, Teale A et al. Magnetic resonance imaging for the head and spine: effective for the clinician or the patient? Br Med J 1991; 302: 78–82.Google Scholar
- Goldin J, Sayre JW. Review: a guide to clinical epidemiology for radiologists: part II statistical analysis. Clin Radiol 1996; 51: 317–24.View ArticlePubMedGoogle Scholar
- Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall, 1991: 404–9.Google Scholar
- Boyd NF, Wolfson C, Moskowitz M. Observer variation in the interpretation of xeromammograms. J Natl Cancer Inst 1982; 68: 357–63.PubMedGoogle Scholar
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74.View ArticlePubMedGoogle Scholar
- Feinstein AR, Sosin DM, Wells CK. The Will Rogers phenomenon. Stage migration and new diagnostic techniques as a source of misleading statistics for survival in cancer. New Engl J Med 1985; 312: 1604–8.View ArticlePubMedGoogle Scholar
- Bosi GJ, Geller NL, Chan EY. Stage migration and the increasing proportion of complete responders in patients with advanced germ cell tumours. Cancer Res 1988; 48: 3524–7.Google Scholar
- Bunt AMG, Hermans J, Smit VTHBM, van de Velde CJH, Fleuren GJ, Bruijn JA. Surgical/pathologic stage migration confounds comparisons of gastric cancer survival rates between Japan and Western Countries. J Clin Oncol 1955; 13: 19–25.Google Scholar