Working Group Meeting February 13-14, 2006

Sheraton Hotel – Atlanta, GA

 

MEETING SUMMARY

View Meeting Agenda

 

Sub-Committee Meetings
Individual meetings of the Methods, Outcomes and Topics subcommittees were held from 8:30 – 9:30 am, allowing the subcommittee chairs and members to finalize recommendations and presentations for the full Working Group.

 

Welcome and Introductions 
The EGAPP Working Group (WG) meeting was convened by Dr. Al Berg , Chair. Working Group members participating were Katrina Armstrong, Ned Calonge, James Haddow, Maxine Hayes, Celia Kaye, Kathryn Phillips, Margaret Piper, Sue Richards, Joan Scott, Ora Strickland, and SteveTeutsch. Consultants present were Toby Citrin, Karen Edwards, Glenn Palomaki and Lynn Short. Dr. Berg reported that he and Linda Bradley had reviewed the Conflict of Interest forms from the Working Group members and determined that there were no conflicts for this meeting.

 

Opening Remarks 
Dr. Muin Khoury, Director of CDC’s Office of Genomics and Disease Prevention praised the cohesiveness and hard work of the Working Group, and expressed CDC’s appreciation to the members. Dr. Bradley introduced new key project consultants and a new member of the EGAPP Steering Committee and liaison from the Secretary’s Advisory Committee on Genetics, Health and Society, Dr. Debra Leonard.

 

Status of Evidence Reviews
Kathleen Szegda presented a brief overview of the status of the first two topics chosen for review. The CYP450 review will have a shorter time frame (about 9 months versus 10-12 months). Both reviews will include assessment of available information on analytic and clinical validity, and the impact of testing and subsequent interventions on important outcomes (clinical utility). Outcomes to be considered will include benefits and harms to the patient and family, as well as societal and public health issues (e.g., availability or access to testing, adequacy of consumer and provider education, resources and infrastructure). The analytic frameworks, key questions, and estimated timelines were discussed. Proposals for CYP450 review have been received from Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Centers (EPCs) and reviewed; the WG discussed the process and criteria for reviews with Dr. Bradley from CDC and Dr. Randhawa from AHRQ. The CYP450 award will be made within a week and the HNPCC award sometime before the end of the year. (Additional information on AHRQ and the EPCs can be found at http://www.ahrq.gov/clinic/epc/epcenters.htm.)

 

Action Items: The working titles for the reviews have been modified to:

 

• Testing for Cytochrome P450 Polymorphisms in Adults with Non-Psychotic Depression Treated with Selective Serotonin Reuptake Inhibitors (SSRIs)
• Screening for Hereditary Nonpolyposis Colorectal Cancer (HNPCC) in Newly Diagnosed Colorectal Cancer Patients and Their Families.

 

Potential Partnership with an Ovarian Cancer Evidence Review funded by CDC’s Division of Cancer Prevention and Control (DCPC)
Dr. Bradley presented the background of this evidence review and the interest of DCPC in having the EGAPP Working Group participate in the review’s Technical Evaluation Panel (TEP) and review the evidence report. The objective is to review evidence for tests that predict risk of ovarian cancer in apparently healthy women, detect ovarian cancer or predict risk of recurrence, or help to tailor treatment. The WG determined that the analytic framework and key questions had relevance to EGAPP, and the evidence review may allow them to make recommendations. WG members will serve on the Ovarian Cancer TEP and follow the progress of the review.

 

Outcomes Subcommittee Report and Discussion
Dr. Botkin presented the lexicon of outcomes that has been stratified into four categories derived from Tatsioni et al., 2005:(1) diagnostic thinking/health information impact, therapeutic choice, patient outcomes impact, and familial and societal impact. It was noted that ELSI did not have a separate category, but was interwoven throughout the other categories. The WG recognized that outcomes would need to be selected from the list for each topic, that literature would probably not be available for many of these domains, and that some outcomes would be less critical for recommendations. Dr. Botkin also presented the specific outcomes that the subcommittee identified as relevant to the CYP450 and HNPCC reviews.

 

Methods Subcommittee Report and Discussion 
The WG needs to define criteria to be used for grading the evidence on analytic validity (AV) and clinical validity (CV). Earlier, the Methods subcommittee considered preliminary drafts that included a suggested hierarchy of study design for AV, and criteria related to study execution and strength of evidence for AV and CV. The CV materials integrate published concepts on assessment of diagnostic tests. 2-6

 

Analytic Validity (AV) 
The WG discussed a proposed hierarchy of study designs that included data from proficiency testing schemes, peer-reviewed reports of formal method comparisons or validation studies, data contained in FDA submissions, and unpublished or non-peer reviewed research, clinical, or manufacturer data. Published, peer-reviewed studies could be considered good, fair or poor quality, depending on study design and execution. FDA submissions for approved products were determined to have undergone a peer review. Some kinds of data were more difficult to characterize or rank, but unpublished and/or non-peer reviewed research, clinical, or manufacturer data would be considered poor quality. Data quality could not be considered good unless there was at least one peer-reviewed report. A reasonable first step would be to see if proficiency testing data is available, and then move to peer-reviewed published studies and FDA submissions. It may also be useful to ask if there is something unique about a given test with regard to the probability of pre- and post-analytic error. The WG considered the value of “evidence by association” (i.e., this method generally performs in this way); this might be considered in the absence of other data. The draft information will be modified and considered further by the Methods subcommittee.

 

Clinical Validity (CV) 
The WG considered a draft Grading Scheme for CV that detailed criteria for Level 1 or good quality evidence, Level 2 or fair quality evidence, and Level 3 or poor quality evidence. The group discussed options available when there is insufficient or no evidence. Will the WG say anything? The use of modeling was proposed and there was agreement that this is likely to be a useful approach. It might then be possible to say that in “this situation” you would need to have “this degree” of predictability. This draft will be modified and considered further by the Methods subcommittee.

 

Clinical Utility (CU) 
The WG agreed that existing methodology could be applied to outcomes, though slight modification may be needed based on different kinds of outcomes that might be considered. There are two parts to evidence, the quality of studies and the certainty of outcome – the USPSTF uses the quality of the study to help arrange the information available and then uses the certainty to make a judgment. The group noted the importance of considering the viability of alternative study designs since randomized controlled trials (RCTs) will not be as commonly seen. A bad RCT is not better than a good cohort study. Quality should be based on the validity of study design and what has been done to minimize bias, not just study type. In spite of sometimes differing understanding of the meaning, the WG decided to keep the term clinical utility because of its general use, but to qualify the term (e.g., clinical utility/outcomes) for clarity.

 

The process of developing recommendations based on the evidence 
Thinking ahead to development of recommendations, there was agreement that the group needs to be clear about who the end-user is and who statements are being written for. If the goal of the group is to guide clinical practice, then the WG must point out the value added of the proposed test. The need to explicitly state the type of benefits and harms being described in the recommendation was discussed – harms can be interpreted differently by different groups. The group acknowledged the need for an education component for clinicians to accompany the recommendation. If this type of targeted review process is used, no evidence is found, and the test is not recommended, it tells the research community that if you want to use the test, you will need to fill in the evidence gaps.

 

Topics Subcommittee Report and Discussion
Dr. Phillips noted the updates to the draft criteria for topic prioritization and selection. She reported that the current process is that all topics received are reviewed for eligibility and added to the list. A short summary is prepared for each that includes the test, disorder, and proposed clinical scenario; each is then added to the spreadsheet and scored based on the criteria. The Topics subcommittee reviews the scoring and recommends to the full WG the topics to move forward for full summaries or fast track review. Since the beginning of the project, 35 topics have been suggested. Currently, 16 short summaries and 13 full summaries have been prepared; 6 topics are pending. The Topics subcommittee has concluded that: 1) short summaries provide sufficient information to help determine which topics to move forward to full summaries; 2) scoring at the short summary stage is a means to an end and there should not be too much emphasis on exact scores; 3) changes to scoring (e.g., review of weighting) should be considered to improve discrimination; 4) contextual information or “implicit criteria” (e.g., inherited versus somatic mutations, test category) are important in the overall view of importance; 5) staff should specifically indicate when information is not available or presented inconsistently; and 6) all full summaries should be put into the updated format that addresses all the criteria. The WG will review full summaries for the February meeting. The WG also considered possible approaches for making the review of potential topics public, possibly by posting on a website. This will also be discussed further in February.

 

“Fast Track” Topic 
The WG discussed the objective of trying to choose a fast-track topic by year’s end. Possible definitions of a “fast-track” review were discussed. For the purposes of the first review, the definition will be that a fast track review is essentially a full review that can be completed in less time because there are few studies. The Topics subcommittee identified several potential topics, including EGFR mutation analysis in non-small cell lung cancer, UGT1A1 testing in colorectal cancer patients, and uPM3 testing to screen for prostate cancer. A quick, preliminary search was conducted to ensure feasibility. The WG requested full summaries on the EGFR and UGT1A1 topics. An email vote will be called after the WG has reviewed the full summaries.

 

Format of Future Meetings
All meetings have been posted, but to date there have been few visitors. However, awareness of EGAPP is growing. CDC staff will more widely disseminate information about future meetings and agendas if the WG is in agreement. Overall, the group was in favor of continuing to have a significant part of meetings as public sessions, but also be able to elect executive session in some cases. The group decided that future meetings will have public sessions that will include presentations, general discussion and some votes, and public comment. Some debate, EPC reports or preliminary votes may be held in executive session. The reasons for executive session include recommendations in progress that may change over time (and might be mistakenly perceived as ‘final’) and topics for which there is intense economic interest in deliberation and outcome. The WG considered a formal process for organizations to submit comments/suggestions that will be implemented immediately. Meeting summaries will continue to be posted on the CDC website, and will include a summary of discussions and any decisions made.

 

For the next meeting:

 

• The Methods subcommittee will consider methods drafts and report/recommendation formats.
• The Topics subcommittee will continue to review topics and work on scoring criteria and weights.
• Staff will prepare full summaries on EGFR and UGT1A1 for review by the full WG and a subsequent email vote on a fast track topic.
• The Outcomes subcommittee will finalize the working draft of the Outcomes lexicon.
• A new Products subcommittee will be formed, set an agenda, and begin their work.

Dr. Berg thanked everyone for their participation and adjourned the meeting.  

 

Footnotes

1. Tatsioni A, Zarin DA, Aronson N, Samson DJ, Flamm CR, Schmid C, Lau J. Changes in Systematic Review of Diagnostic Technologies. Ann Intern Med 2005;142:1048-55.
2. Levels of Evidence and Grades of Recommendation. http://www.cebm.net/levels_of_evidence.asp.
3. Harris R, Helfand M, Woolf S, Lohr K, Mulrow C, Teutsch S, Atkins D. Current Methods of the U.S. Preventive Services Task Force: A Review of the Process. Am J Prev Med 2001; 20:21-35.
4. AHRQ. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment 2002; 47; 1-11.
5. Ebell M, Siwek J, Weiss B, Woolf S, Susman J, Ewigman B, Bowman M. Strength of Recommendation Taxonomy (SORT): A Patient-Centered Approach to Grading Evidence in the Medical Literature. JABFP 2004; 17: 59-67.
6. Whiting P, Rutjes AWS, Reitsma JB, Bossuyt PMM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies on diagnostic accuracy included in systematic reviews. BMC Med Research Method 2003;3:25-38.