Genetic Test Evaluation: Information Needs of Clinicians, Policy-Makers and the Public

by Wylie Burke¹, David Atkins², Marta Gwinn³, Alan Guttmacher⁴, James Haddow⁵, Joseph Lau⁶, Glenn Palomaki5, Nancy Press⁷, C. Sue Richards⁸, Louise Wideroff⁹, Georgia L. Wiesner¹⁰

Am J Epidemiol. 2002 Aug 15;156(4):311-8

¹ Epidemiology Group, Department of Medicine & Therapeutics, University of Aberdeen, Scotland
² Sunrise Medical Laboratory, Hauppauge, New York
³ University of Texas Health Science Center at Houston, Houston, TX
⁴ Office of Genetics & Disease Prevention, Centers for Disease Control & Prevention, Atlanta, GA
⁵ Graduate School of Public Health, Department of Epidemiology, Pittsburgh, PA
⁶ National Heart, Lung and Blood Institute, Bethesda, MD
⁷ Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD
⁸ Division of Clinical Care Research, New England Medical Center, Boston, MA
⁹ Epidemiology Program Office, Centers for Disease Control & Prevention, Atlanta, GA
¹⁰ Ospedale Policlinico IRCCS, Milano, Italy

This paper represents consensus from the CDC-NIH Human Genome Epidemiology Workshop, January, 2001.

Introduction

Understanding the genetics of human health typically begins with the identification of genetic variants associated with specific diseases. Epidemiological studies play a key role in identifying these associations. Assigning causality can be difficult, however, because of the multifactorial nature of most diseases. Even when a genetic mutation conferring increased risk is present, health outcomes may be influenced by a variety of environmental exposures, behaviors and other genes, and interaction between some or all etiologic factors may occur. The analytic complexity posed by multifactorial disease is the subject of the two other papers in this series (1,2). The present paper considers the information needs that arise when a genetic risk is sufficiently established that its assessment is considered in clinical or public health practice – that is, the point at which information about a gene-disease association becomes the basis for a genetic test.

All of the issues discussed by Little et al. (1)and Hunter et al (2), concerning the interpretation of gene-disease associations, and gene-gene and gene-environment interactions, remain relevant at this transition point, because they influence the interpretation of test results. However, a new issue arises when a genetic test is considered for clinical or public health use: whether the test provides a health benefit.

Because benefit can be evaluated only in the context of specific health outcomes, the starting point for considering the use of a genetic test is a well-defined clinical problem, for which the test is expected to improve care. The test may help direct work-up or management of a clinical problem, identify candidates for specific interventions, or provide diagnostic or prognostic information. In particular, genetic testing is expected to provide unparalleled new opportunities for the prevention of disease and adverse drug reactions, through the identification of people with genetic susceptibilities (3,4).

Over the next decade a growing number of genetic tests will be marketed to clinicians, as a result of expanding genetic knowledge. Many of these new tests will use DNA-based technology, but any laboratory test used primarily to identify an inherited condition is considered a genetic test (5). More than eight hundred genetic tests are currently available or in research development (GeneTests-GeneClinics; http://www.geneclinics.org). Although most currently available tests are for rare diseases, tests to identify inherited risk for common diseases such as breast and colorectal cancer, thromboembolism, Alzheimer Disease, and coronary heart disease have been developed.

As new genetic tests become available, policy makers, clinicians, and the public will need to make decisions about test use. New tests are likely to vary in their predictive value, their potential to direct prevention or treatment efforts, and their personal and social consequences. Therefore, the task of determining the appropriate use for a given test will require careful consideration of a variety of factors. In developing practice guidelines, different professional organizations, agencies and health care systems will utilize a range of methodologies to evaluate evidence and reach final decisions about the indications for test use
(6-8). Guidelines procedures also vary in their consideration of costs and other social issues, in part because of differences in health care priorities and resources in different health care settings. However, all practice guidelines procedures require clear and accurate information about the test under consideration. In addition, the appropriate implementation of most practice guidelines involves clinical judgment and consideration of patient preferences; thus clinicians and patients also need accessible and accurate information about genetic test properties. To aid policy makers, clinicians and the public in the prudent use of new genetic tests, we propose a standardized approach to reporting the data defining test properties. Our discussion of the issues to be addressed in the reporting process is illustrated by examples of genetic tests with high and low predictive value.

PKU: High expectations for improved prevention

Newborn screening for phenylketonuria (PKU) represents a model for the use of genetic testing for disease prevention. This public health genetics program identifies newborns with PKU, a rare genetic condition in which failure to metabolize dietary phenylalanine leads to severe mental retardation. When affected newborns are provided with a phenylalanine-restricted, tyrosine-supplemented d, mental retardation can be prevented. To a large extent, the current enthusiasm for genetic testing derives from the belief that similar benefits may occur with the use of predictive genetic tests for a wide array of other diseases.

The success of newborn screening for PKU can be attributed to four factors: a reliable and accurate screening process; a readily identifiable population for screening; an effective treatment for those who test positive; and timely initiation of treatment (9).The current PKU screening protocol, which includes follow-up testing for confirmation after an initial elevated phenylalanine level, leads to highly sensitive and specific identification of children with PKU. However, initial screening efforts did not have this level of accuracy. In the initial phases of PKU screening, some children with moderate phenylalanine elevations were classified as affected and suffered adverse consequences from a phenylalanine restricted diet (10).

Further, although the phenylalanine restricted diet is effective in preventing mental retardation, it is not without burdens. Children must be taught to avoid many common foods and to adhere to a diet that is not palatable to most people. Social burdens are involved; a child with PKU must be taught to avoid the hamburgers and ice cream that may be offered at a friend’s house and to handle the inevitable explanations. Education and follow-up are needed to enhance compliance, with special attention to the transition to self-management that occurs in adolescence. In addition, subsidized programs for disadvantaged families are necessary to ensure that the diet is available to all newborns who test positive.

Finally, it was not until newborn screening for PKU had been instituted that an additional complication of PKU was identified: microcephaly in children of mothers affected with PKU whose phenylalanine levels were elevated during pregnancy. This outcome led to recognition of the need for tight dietary control before and during pregnancy in women with PKU. Similarly, on-going follow-up has demonstrated that all patients with PKU benefit from continued dietary control in adulthood, illustrating the long-term social consequences of this disorder.

These experiences emphasize that data on a broad range of issues is relevant to the evaluation of a genetic test. These include not only the expected clinical measures – the reliability of the laboratory measurement, the predictive value of the test and the effectiveness of medical interventions – but also social issues such as access to care and the personal and familial implications of testing and interventions. The PKU experience also underscores the need for on-going data collection after a test is introduced, to identify unanticipated consequences of the testing process.

Genetic tests with limited predictive value: Factor V Leiden and venous thromboembolism
Can the PKU experience be replicated with other genetic tests? It is unlikely that genetic tests for common disorders will have the predictive value of PKU testing or lead to interventions that are as definitive. This is because most common disorders are multifactorial: many different genetic variants may contribute to risk, in conjunction with environmental risk factors. As a result, genetic variants associated with common diseases usually indicate increased risk rather than certainty of future disease, and the interaction of genetic and non-genetic factors is often important in determining the risk status of a particular individual. In addition, the experience to date suggests that genetic tests associated with an increased risk for common diseases may be offered for clinical use well before outcome studies are available to determine the benefits of interventions for those with genetic susceptibility (11,12).

Factor V Leiden, the DNA-based test most commonly ordered in the US, offers an example to illustrate these issues. This genetic trait is a variant of the factor V gene, which codes for a protein involved in clot formation. It is present in approximately 5% of European Americans, 2% of Hispanic Americans, and 1% of African and Asian Americans (13). Many studies, done predominantly in populations of European descent, have now documented its association with an increased risk of venous thromboembolism (VT), often in the setting of other circumstances that affect risk, including pregnancy, use of oral contraceptives and surgery (13-17). Estimates of the increased risk for VT among carriers of factor V Leiden vary from 2- to 8-fold (17-18).

Factor V Leiden is one of several genetic factors that contribute to risk for VT, (17) and interactions between genetic and non-genetic risk factors for VT are well-established (17-19) For example, individuals carrying both factor V Leiden and the 202210A variant in the prothrombin gene have a substantially higher risk of VT than carriers of either trait alone (18,20,21). In more than half of a series of families with antithrombin III deficiency, another genetic risk factor, the high risk of VT was due to the additional presence of factor V Leiden or the prothrombin variant (22). Thus, when VT occurs in the presence of factor V Leiden it often – perhaps always – represents the effect of interactions with one or more other genetic and non-genetic risk factors (17). Studies of the annual incidence of VT events in people with factor V Leiden suggest that the cumulative risk is in the range of 10% to 25% (14,19,21).

Although factor V Leiden is well established as a risk factor, its implications for clinical management are far from clear. Several rationales for factor V Leiden screening have been proposed. Carriers of the variant might benefit from avoiding oral contraception (15,21)or post-menopausal estrogen (24). Anticoagulant prophylaxis might reduce the risk of VT when transient risk factors such as surgery, pregnancy or prolonged bed rest, occur (17,21,23). However, these interventions all pose potential risks as well as benefits, and none has been systematically studied in people with factor V Leiden.

These various possibilities demonstrate the importance of focusing on a defined clinical problem and setting when determining the evidence needed to assess test use. Studies assessing different test purposes require different testing protocols, study populations and outcome measures. For example, an assessment of the use of factor V Leiden testing prior to oral contraceptive use would require a population of women receiving contraceptives, and would need to evaluate both contraceptive-related thrombotic risk and the outcome of alternative methods of contraception. Conversely, the assessment of testing to determine the appropriate thrombosis prophylaxis prior to surgery would rely on the measurement of post-operative outcomes in surgical patients using specified prophylaxis protocols. Each study would require appropriate control populations and measurement of other factors associated with risk of VT. Without such studies the clinical utility of testing for factor V Leiden is called into question.

Developing standardized approaches to presenting evidence
Both the PKU and factor V Leiden examples demonstrate the importance of presenting data about genetic tests in formats that allow the specific clinical outcomes of tests to be evaluated. Data to assess genetic tests for this purpose need to address three general categories of test performance:(5)

1. Analytic Validity

   The term analytic validity refers to the accuracy with which a particular genetic characteristic (for example, a DNA sequence variant) can be identified in a given laboratory test. One way to report analytic validity is in terms of the test’s sensitivity and specificity for the genetic variant in question. Most genetic variants can be tested by a variety of protocols, and a number of technical issues arise in evaluating analytic validity. These include the assay chosen, the reliability of the assay, the degree to which reliability varies from laboratory to laboratory, and the complexity of test interpretation. Thus, an oligonucleotide probe for a single nucleotide sequence variant is a simpler test than a linkage analysis. In the latter, accuracy of the test is based on the accuracy with which samples and medical history are collected from family members, as well as on technical aspects of the testing process in the laboratory.

   Often, few published data are available concerning the analytic validity of a test, and it may be difficult to judge whether different studies of the same genetic trait used comparable laboratory methods. For example, some studies of factor V Leiden utilize a DNA-based assay while others utilize a functional assay, with differences in sensitivity and specificity. An adequate description of the analytic validity of a test requires systematic collection of data for this purpose, using defined populations with known genotypes (Table 1). Proficiency testing, such as that performed under the auspices of the American College of Medical Genetics and the College of American Pathologists (ACMG/CAP), provides one source of objective data on analytic validity, but proficiency testing usually involves small numbers and few studies of normal genotypes. The scarcity of published studies on analytic validity is an important limitation in the available evidence about genetic tests.

2. Clinical Validity

   The term clinical validity describes the accuracy with which a test predicts a particular clinical outcome. When a test is used diagnostically, clinical validity measures the association of the test with the disorder. When a test is used to identify genetic susceptibility, clinical validity measures the accuracy with which it predicts a future clinical outcome. This test property corresponds to the gene-disease association measured in epidemiological studies (1). A useful way to present clinical validity data for clinicians is in terms of the positive and negative predictive value of the test for the occurrence of disease within a defined population.

   Clinical validity is often uncertain for new genetic tests. Estimates may be based on studies in limited and potentially biased populations. Uncertainty about clinical validity is an important consideration for policy-makers, clinicians and patients, particularly when they are deciding whether the risk information provided by the test provides a sufficient rationale for the use of interventions with putative but unproven benefits.

   Mutations associated with disease susceptibility are often first defined in “high risk” families, characterized by multiple affected family members, and these studies may over-estimate risk. The BRCA1 and BRCA2 genes, for example, were first identified in families selected for the presence of breast cancer under age 60 or ovarian cancer in four or more family members (25). The lifetime risk of breast cancer is estimated to be about 85% in mutation carriers from these high risk families. But studies using less selected populations have estimated the lifetime risk associated with BRCA1/2 mutations to be 36%-56% (26-29). These observations suggest that high risk families represent the severe end of a spectrum of risk associated with BRCA1/2 mutations and that other factors, both genetic and non-genetic, may modify risk (30). Thus, variation in clinical validity may be due to complex gene-gene and gene-environment interactions, (2) with the result that the predictive value of many genetic tests may remain imprecise even when all appropriate studies have been completed.

   Important variables in evaluating evidence about clinical validity include the size and selection criteria for the study population, the type of test used (and its analytic validity), study design, and clinical and other end points measured (Table 1). Careful definition of the measures used to define the clinical outcome is needed. Studies will provide more convincing results when they also include measurement of non-genetic factors that contribute to the clinical outcome. Key methodological considerations include the comparability of case and control populations (where a case/control design is used), whether the interpretation of end-points was blinded and whether negative results were verified.

   Many genetic variants affect more than one clinical outcome; for example, Factor V Leiden is associated with both VT and pregnancy loss (31), and some genetic variants increase risk for two or more different cancers (25).Multiple disease end-points need to be considered when studying such variants; clinical validity may be high for one end-point and low for another.

3. Clinical Utility

   Clinical utility refers to the likelihood that the test will lead to an improved health outcome. Measurement of this characteristic requires evaluation of outcomes associated with testing and clinical interventions. Central questions are the effectiveness of interventions available for people who test positive and the social consequences of testing for people with positive and negative test results. Standardized approaches are available to assess the quality of evidence concerning interventions (32).

   As with information about clinical validity, knowledge about optimal prevention strategies is accumulated incrementally. As is usual with emerging technology, people most genetically susceptible are initially offered interventions based on clinical reasoning or extrapolation from data in other populations – for example, early initiation of mammography screening in women with BRCA1/2 mutations. The risks and benefits of these interventions can be understood only after systematic observation, in the form of well-designed controlled trials, cohort or case-control studies. Because most genetic risk factors, even for common diseases, occur in a small percentage of the population – e.g., the 1%-5% prevalence of Factor V Leiden – , sample sizes of genetically susceptible subjects are often small. Definitive understanding of clinical utility may be dependent on reliable methods for the pooled analysis of different studies.

   Important variables in evaluating evidence about clinical utility are the size and selection criteria for the study population, the type of laboratory assay and intervention used, and the study design (Table 1). The health benefit proposed for a test will determine some of the outcome measures; others are determined by hypotheses concerning the harms of the testing process. As with clinical validity studies, the comparability of the cases and controls is an important issue, when a case/control design is used. If an experimental design is used, elements of the study methodology – for example, randomization strategy – are important. The methods for measuring clinical and other end-points should be specified, as well as whether negative results were verified.

   Because genetic variants may affect multiple different clinical end-points, a test that has high clinical utility for one end-point may also provide extraneous information of little utility for another. For example, a person with a mutation predisposing to hereditary nonpolyposis colorectal cancer (HNPCC) faces a high lifetime risk of colorectal cancer and is likely to benefit from early initiation of regular colonoscopy, as a means to reduce colorectal cancer mortality (33). However, HNPCC is also associated with increased risk for endometrial, ureteral tract and upper gastrointestinal cancers; methods of screening for endometrial cancer are of unproven efficacy, and no preventive care is available for the other cancers associated with HNPCC (34).

Summary

New techniques to organize and disseminate evidence about genetic tests are an important step in support of the rational and prudent use of this new technology. To be effective, evaluation methods must be as complete and unbiased as possible. Clinicians, guidelines panels and other policy-makers must consider genetic tests in light of alternative approaches to care; that is, the health benefits and other outcomes of a genetic test must be evaluated compared to the outcomes that could be obtained in the absence of genetic testing. To make this comparison, adequate evidence concerning the analytic validity, clinical validity and clinical utility of genetic tests is needed, including appropriate control populations and, ideally, measurement of social outcomes.

This ambitious research agenda is unlikely to be achieved rapidly or completely for most genetic tests. Even when all appropriate research is completed, the predictive value of many genetic tests may remain imprecise and the clinical utility may be variable. As a result, methods to disseminate information about genetic tests represent an important strategy for ensuring appropriate test use. Following criteria for the objective assessment of test properties, reports should be structured to enable policy makers, clinicians and the public to identify quickly what is known and not known about analytic validity, clinical validity and clinical utility, so that uncertainties can be taken into account when genetic tests are considered for use.

Acknowledgements

This paper represents consensus from the Centers for Disease Control and Prevention-National Institutes of Health Human Genome Epidemiology Workshop, January 2001.

Table 1: Presenting Data About Genetic Tests

Table 2: Evidence Table for a Hypothetical Study of Clinical Utility

References

 Top of Page