Skip directly to search Skip directly to A to Z list Skip directly to navigation Skip directly to page options Skip directly to site content

EGAPP™ Working Group Methods Summary

This website is archived for historical purposes and is no longer being maintained or updated.

The independent Evaluation of Genomic Applications in Practice and Prevention (EGAPP™) Working Group reviews the scientific evidence for selected genetic tests and develops recommendation statements about the appropriate use of these tests. The following tables provide summarized information about the methods that the EGAPP™ Working Group uses to conduct this work. These methods have also been published in January 2009 issue of Genetics in Medicine.
 

Table 1: Test Applications

This table describes categories of genetic test applications and some characteristics of how clinical validity and clinical utility are assessed for each.

Application Clinical Validity Clinical Utility
Diagnosis (symptomatic patient) Association of marker with disorder
  • Improved clinical outcomes - health outcomes based on diagnosis and subsequent intervention or treatment
  • Availability of information useful for personal or clinical decision-making
  • End of diagnostic odyssey
Disease Screening (asymptomatic patient) Association of marker with disorder Improved health outcome based on early intervention for screen positive individuals to identify a disorder for which there is intervention or treatment, or provision of information useful for personal or clinical decision making
Risk assessment/susceptibility Association of marker with future disorder (consider possible effect of penetrance) Improved health outcomes based on prevention or early detection strategies
Prognostic Association of marker with natural history benchmarks of the disorder Improved health outcomes, or outcomes of value to patients, based on changes in patient management
Pharmacogenomic Predicting treatment response or adverse events Association of marker with a phenotype/metabolic state that relates to drug efficacy or adverse drug reactions Improved health outcomes or adherence based on drug selection or dosage

 Top of Page

Table 2: Components of Evidence Evaluation

Because of the newness of the field of genetic testing, direct evidence to answer an overarching question about the effectiveness and value of testing is rarely available. Therefore, the EGAPP™ Working Group evaluated evidence in three key areas to construct a chain of evidence to address the overarching question.

Analytic Validity
(Technical Performance)		 Clinical Validity
(Strength of Clinical Correlation)		 Clinical Utility
(Impact on Health Outcomes)

Tests ability to accurately and reliably measure analyte or genotype of interest.

  • Sensitivity
  • Specificity
  • Assay robustness
  • Quality control

Test’s ability to accurately and reliably identify or predict the disorder of interest.

  • Sensitivity
  • Specificity
  • Positive Predictive Value
  • Negative Predictive Value

Likelihood that using the test to guide management will significantly improve health-related outcomes.

  • Benefits vs. Harms
  • Added value compared to not using test
  • Effectiveness
  • Efficacy

 Top of Page

 

Table 3: Quality of Evidence

This table describes how the quality of evidence was graded in terms of its adequacy to address the key questions of each of the evidence components: analytic validity, clinical validity, and clinical utility.

Note: The “levels” used in ranking studies are described in Table 4.

Quality of Evidence
Adequacy of Information to Answer Key Questions Analytic Validity Clinical Validity Clinical Utility
Convincing

Studies that provide confident estimates of analytic Sensitivity and specificity using intendedsample types from representative populations

  • Two or more Level 1 or 2 studies that are generalizable, have a sufficient number and distribution of challenges, and report consistent results
  • One Level 1 or 2 study that is generalizable and has an appropriate number and distribution of challenges

Well-designed and conducted studies in representative population(s) that measure the strength of association between a genotype or biomarkerand a specific and well-defined disease or phenotype

  • Systematic review/meta-analysis of Level 1 studies with homogeneity
  • Validated Clinical Decision Rule
  • High quality Level 1 cohort study

Well-designed and conducted studies in representative population(s)that assess specified health outcomes

  • Systematic review/meta-analysis of randomized controlled trials showing consistency in results
  • At least one large randomized controlled trial (Level 2)
Adequate
  • Two or more Level 1 or 2 studies that
    • Lack the appropriate number and/or distribution of challenges
    • Are consistent, but not generalizable.
  • Modeling showing that lower quality (Level 3, 4) studies may be acceptable for a specific well-defined clinical scenario
  • Systematic review of lower quality studies
  • Review of Level 1 or 2 studies with heterogeneity
  • Case-control study with good reference standards
  • Unvalidated Clinical Decision Rule (Level 2)
  • Systematic review with heterogeneity
  • One or more controlled trials with out randomization (Level 3)
  • Systematic review of Level 3 cohort studies with consistent results
Inadequate
  • Combinations of higher quality studies that show important unexplained inconsistencies
  • One or more lower quality studies (Level 3 or 4)
  • Expert opinion
  • Single case-control study
    • Nonconsecutive cases
    • Lacks consisitently applied reference standards
  • Single Level 2 or 3 cohort/case-control study
    • Reference standard defined by the test or not used systematically
    • Study not blinded
  • Level 4 data
  • Systematic review of Level 3 quality studies or studies with heterogeneity
  • Single Level 3 cohort or case-control study
  • Level 4 data

 Top of Page

Table 4: Ranking of Data Sources/Study Designs for Components of Evaluation

This table illustrates how data sources and study designs were ranked in order to assess the quality of evidence for each component of the evidence evaluation.

Levela Analytic Validity Clinical Validity Clinical Utility
1
  • Collaborative study using a large panel of well characterized samples
  • Summary data from well-designed external proficiency testing schemes or interlaboratory comparison programs
  • Well designed longitudinal cohort studies
  • Validated clinical decision ruleb
 Meta-analysis of randomized controlled trials (RCT)
2
  • Other data from proficiency testing chemes
  • Well designed peer-reviewed studies (e.g., method comparisons, validation studies)
  • Expert panel reviewed FDA summaries
  • Well designed case-control studies
  • A single randomized controlled trial
3
  • Less well designed peer-reviewed studies
  • Lower quality case-control and cross-sectional studies
  • Unvalidated clinical decision ruleb
  • Controlled trial without randomization
  • Cohort or case-control study
4
  • Unpublished and/or non-peer reviewed research, clinical laboratory, or manufacturer data
  • Studies on performance of the same basic methodology, but used to test for a different target
  • Case Series
  • Unpublished and/or non-peer reviewed research, clinical laboratory or manufacturer data
  • Consensus guidelines
  • Expert opinion
  • Case Series
  • Unpublished and/or non-peer reviewed studies
  • Clinical laboratory or manufacturer data
  • Consensus guidelines
  • Expert opinion

aHighest level is 1.

bA clinical decision rule is an algorithm leading to result categorization. It can also be defined as a clinical tool that quantifies the contributions made by different variables (e.g., test result, family history) in order to determine classification/interpretation of a test result (e.g., for diagnosis, prognosis, therapeutic response) in situations requiring complex decision-making.

 Top of Page

Table 5: Recommendation Classification

This table demonstrates the possible recommendations derived from the evaluation of the evidence components, the overall level of certainty of net health benefits, and contextual factors.

Level of Certainty Recommendation
High or Moderate

Recommend for . . .
. . . if the magnitude of net benefit is substantial, moderate, or small, unless additional considerations warrant caution.

Recommend against . . .
. . . if the magnitude of net benefit is zero or there are net harms.
Low Insufficient evidence . . .
. . . if the evidence for clinical utility or clinical validity is insufficient in quantity or quality to support conclusions or make a recommendation.

Disclaimer: 
Our office does not offer medical advice to individuals. If you have specific concerns about your health, please discuss them with your doctor.

 Top of Page

Top