Assessment of ESSENCE Performance for Influenza-Like Illness Surveillance After an Influenza Outbreak --- U.S. Air Force Academy, Colorado, 2009

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE), version II, designed by the Johns Hopkins University Applied Physics Laboratory and the U.S. Department of Defense (DoD), is an Internet-based syndromic disease surveillance system used by civilian and military health departments (1). ESSENCE was designed to increase the timeliness of outbreak detection, serving as an early warning system and providing opportunities to prevent and control the spread of infection. After a 2009 pandemic influenza A (H1N1) outbreak at the U.S. Air Force (USAF) Academy in Colorado, CDC was invited to conduct an evaluation of the ESSENCE influenza-like illness (ILI) surveillance system to assess its performance during the outbreak (2,3). Medical records at the USAF Academy clinics from June 25 through July 8, 2009, the period of the outbreak, were reviewed. This report summarizes the results of the evaluation, which demonstrated strengths in data quality, flexibility, and representativeness; however, ESSENCE was not useful for detecting or monitoring the H1N1 outbreak because of its lack of timeliness (1--3 day delay), inadequate sensitivity (71.4%), and poor predictive value positive (PVP) (31.8%) for identifying ILI cases. In this localized, single-source outbreak, ESSENCE did not serve as an early warning system for an emerging infectious disease and did not detect the outbreak soon enough to institute prevention and control measures that might have slowed the spread of infection. More frequent Internet data transmissions from the clinics to the ESSENCE server could improve timeliness, and PVP could be enhanced by including measured body temperature in the ESSENCE ILI case definition.

The utility of syndromic disease surveillance for early outbreak detection and improvement of public health response remains controversial (4−7). A survey of U.S. health departments indicated that the most common application for syndromic surveillance was to monitor the start and stop of the annual influenza season, but that it was less useful for local outbreak detection (4). Other studies found that syndromic surveillance has been useful to identify localized respiratory, dermatologic, and gastrointestinal disease outbreaks (5−7).

During June 25--July 24, 2009, an H1N1 outbreak occurred at the USAF Academy in Colorado, with 134 cases confirmed among a population of USAF 1,376 basic cadet trainees (3). Although ESSENCE is used at the USAF Academy, public health officials became aware of the outbreak before ESSENCE indicated the increase in ILI cases. After this large outbreak, the USAF Academy invited CDC to evaluate ESSENCE as an ILI surveillance system.

The U.S. military has used ESSENCE since 2003 to detect and monitor disease outbreaks. DoD provides an annual budget for system maintenance and development and releases updated versions of the system as surveillance needs change. ESSENCE identifies patients based on provider-assigned International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes that are entered into the electronic medical record (1). The ESSENCE ICD-9-CM code set for ILI* was established based on an analysis of medical records and respiratory specimens to determine the codes that most accurately represent ILI (8,9). By regular secure data transmissions via the Internet, the ESSENCE server identifies new cases and, using temporal algorithms that predict expected daily fluctuation, determines whether an increasing trend has occurred, indicating a possible outbreak (1,8,9). Raw data and aggregate reports, in the form of line graphs, are available via a password-protected ESSENCE website. An increase in syndromic cases above predicted thresholds is highlighted as a color-coded alert. Privacy and confidentiality are maintained with patient identification numbers and annual information protection training for users. In 2003, using past military and civilian data, an initial evaluation determined that ESSENCE detected eight of eight respiratory disease outbreaks within an average of 1 day after the event (1).

CDC's Updated Guidelines for Evaluating Public Health Surveillance Systems was used to assess the usefulness, simplicity, flexibility, data quality, acceptability, representativeness, timeliness, stability, sensitivity, and PVP of ESSENCE for the USAF (2). To determine sensitivity and PVP, medical record data from ILI case-patient visits at the USAF Academy acute care and cadet clinics during June 25--July 8, 2009, were collected. For the medical record review, ILI was defined as measured temperature ≥100.0°F (≥37.8°C) and cough or sore throat. Medical record data collection included cough, sore throat, measured temperature, and the results of respiratory disease laboratory tests for influenza A, influenza B, H1N1, adenovirus, and group A streptococcus bacteria. Patients from the same period were identified on the ESSENCE ILI website to collect the ICD-9-CM codes. Sensitivity and PVP were calculated using 1) medical record--confirmed ILI, as defined and 2) laboratory confirmation of a respiratory infection, as criterion standards. USAF Academy, USAF School of Aerospace Medicine (USAFSAM), DoD, and CDC staff members who used ESSENCE daily were interviewed to assess the remaining evaluation criteria.

This evaluation found that the usefulness of ESSENCE varied by user. CDC, which used the aggregate USAF and DoD data, found ESSENCE useful to monitor national syndromic disease activity, and USAFSAM staff members found it useful to monitor disease activity at each base. The USAF Academy indicated ESSENCE was useful to monitor the local influenza season and determine syndromic baselines. Although users investigated ESSENCE alerts and worrying trends, most alerts and trends were time-consuming false alarms that revealed normal disease variations.

Regarding simplicity of operations and structure, the ESSENCE website's aggregate reports, line graphs, and color-coded alerts were easy to comprehend. ESSENCE's flexibility to adapt was demonstrated by the updated versions released by DoD based on user feedback and changes in surveillance needs, and the raw data query functions available to ESSENCE users. Data quality, or the completeness and validity of the data, was established by extracting demographic and medical information from official DoD systems, and scheduling automated batched data transmissions to the ESSENCE server at night, during periods of lower Internet usage to reduce transmission interruptions. For acceptability, or the willingness to participate in the surveillance system, data transmission to the ESSENCE server was automated, and personnel at the Academy and USAFSAM had passwords, although cumbersome to attain and maintain, to use ESSENCE in their daily operations. ESSENCE was determined to have a high degree of representativeness, in that it included all DoD beneficiaries visiting all USAF outpatient clinics, and thus it reported all medical events. Timeliness, or the time of the clinic visit to the time the information appeared on the ESSENCE website, was 1--3 days (10). Stability, or the reliability and availability of the system, was maintained by the annual DOD budget and the infrequent occasions when ESSENCE was unavailable to the user.

Of the 540 medical records reviewed to assess sensitivity and PVP, 189 had a laboratory test result. Compared with medical record--confirmed ILI, ESSENCE ILI sensitivity was calculated at 71.4% and PVP at 31.8% (Table). Compared with laboratory-confirmed respiratory infections, ESSENCE ILI sensitivity and PVP were 78.6% and 49.5%, respectively. When the evaluators added a documented, measured body temperature of ≥100.0°F (≥37.8°C) to the ESSENCE ILI case definition, the ESSENCE ILI sensitivity, compared with medical records, remained the same, but PVP increased to 95.5%; however, when compared with laboratory confirmation, ILI sensitivity was 65.7%, and PVP was 66.7% (Table).

Reported by

C Witkop, MD, USAF Academy, Colorado. M Duffy, DVM, USAF School of Aerospace Medicine, Wright-Patterson Air Force Base, Ohio. L Cohen, MD, Scientific Education and Professional Development Program Office, Office of Surveillance, Epidemiology, and Laboratory Svcs; D Fishbein, MD, Div of Global Migration and Quarantine, National Center for Emerging and Zoonotic Infectious Diseases; M Selent,* DVM, EIS Officer, CDC. *Corresponding contributor: Monica Selent, Div of Global Migration and Quarantine, National Center for Emerging and Zoonotic Infectious Disease, CDC, 404-520-2332, mselent@cdc.gov.

Editorial Note

This evaluation found that the major strength of ESSENCE ILI surveillance was its usefulness for monitoring annual seasonal influenza activity. Other strengths included simplicity, flexibility, data quality, representativeness, and stability. Weaknesses included low PVP, lack of timeliness, and limited usefulness to detect and monitor an ILI outbreak.

Retrospectively, ESSENCE showed an increasing ILI trend 2--4 days before an Academy mass gathering; however, the combination of the ESSENCE time delay, occurrence of the gathering over a holiday weekend, and short incubation period of H1N1 meant the increasing trend was not detected in time to institute preventive measures. To improve timeliness, medical data transmission could be scheduled in smaller, more frequent batches throughout the day so that changing trends would appear on the ESSENCE website sooner.

After USAFSAM judged that the ILI PVP of the surveillance system was too low to distinguish actual outbreaks, the addition of a measured body temperature ≥100.0°F (≥37.8°C) to the ESSENCE case definition was evaluated to determine whether PVP, and potentially sensitivity, could be improved. Compared with medical record--confirmed ILI and laboratory-confirmed respiratory infections as criterion standards, PVP did increase with the addition. An independent study using only laboratory confirmation and ICD-9-CM--based ILI surveillance also found that PVP increased by adding measured body temperature (10). The large PVP increase with medical record confirmation was attributed to the low number of ESSENCE ILI cases with an elevated temperature at the clinic, potentially resulting from antipyretic use or actual afebrile infection. When compared with laboratory-confirmed respiratory infections, sensitivity decreased. However, with medical record confirmation, sensitivity stayed the same, because all the ILI cases still had an elevated temperature, per the medical record case definition, and an ILI ICD-9-CM code, per the ESSENCE case definition. Despite the improvement in PVP by adding a measured body temperature to the ESSENCE case definition, the potential loss in sensitivity might reduce the ability to detect actual ILI outbreaks. Users need to determine if this loss is acceptable for their purposes.

The findings in this report are subject to at least three limitations. First, as new data arrive in ESSENCE, the web page does not record the date additional case-patients appeared. Therefore, evaluators could only estimate when ESSENCE issued an alert to the ILI outbreak, based on historical documentation. Second, this evaluation collected data from only one outbreak at one USAF base. Additional outbreak analyses from other USAF bases are needed to judge the effectiveness of ESSENCE as an early-warning outbreak system for the USAF. Finally, the results of this evaluation are not generalizable to the other military services or civilian public health agencies, which might use ESSENCE differently.

This evaluation showed that, despite strengths in data quality, flexibility, and representativeness, ESSENCE did not serve as an early warning system for an emerging infectious disease during a localized, single-source outbreak, and did not detect the outbreak soon enough to allow prevention and control measures to be instituted. For enhanced outbreak detection and monitoring, more frequent Internet data transmissions would improve ESSENCE's timeliness. Additionally, the inclusion of measured body temperature in the ESSENCE ILI case definition could improve PVP, but with a possible loss in sensitivity resulting from exclusion of afebrile cases. As the strengths, weaknesses, and limitations of ILI surveillance as an early warning system for emerging infectious disease become better understood, future development should investigate how informatics and information technology can overcome ILI surveillance weaknesses.

Acknowledgments

This report is based, in part, on contributions by K Cox, MD, US Army Center for Health Promotion and Preventive Medicine; J Collins, R Devine, A Cox, A Owens, M Green, US Air Force Academy; and C Hales, PhD, N Molinari, PhD, N Megateli-Das, MS, P Szymanowski, MPH, C Adams, and J Herrera, CDC.

References

1. CDC. ESSENCE II and the framework for evaluating syndromic surveillance systems. MMWR 2004;53(Suppl):159--65.
2. CDC. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR 2001;50(No. RR-13).
3. Witkop CT, Duffy MR, Macias EA, et al. Novel influenza A (H1N1) outbreak at the U.S. Air Force Academy. Am J Prev Med 2010;38:121--6.
4. Buehler JW, Sonticker A, Paladini M, Soper P, Mostashari F. Syndromic surveillance practices in the United States: findings from a survey of state, territorial, and selected local health departments. Adv Dis Surveill 2008;6:1--20.
5. Ang BCH, Chen MIC, Goh TLH, Ng YY, Fan SW. An assessment of electronic captured data in the Patient Care Enhancement System (PACES) for syndromic surveillance. Ann Acad Med Singapore 2005;34:539--43.
6. CDC. Comparison of syndromic surveillance and a sentinel provider system in detecting an influenza outbreak---Denver, Colorado, 2003. MMWR 2005;54(Suppl):151--6.
7. Lewis MD, Pavlin JA, Boomsma LG, Yevgeniy E, Kelley PW. Disease outbreak detection system using syndromic data in the greater Washington DC area. Am J Prev Med 2002;23:180--6
8. Marsden-Haug N, Foster VB, Gould PL, Elbert E, Wang H, Pavlin JA. Code-based syndromic surveillance for influenza-like illness by International Classification of Diseases, Ninth Revision. Emerg Infect Dis 2007;13:207--16.
9. Batancourt JA, Hakre S, Polyak CS, Pavlin JA. Evaluation of ICD-9 codes for syndromic surveillance in the electronic surveillance system for the early notification of community-based epidemics. Mil Med 2007;172:346--52.
10. Pattie DC, Atherton MJ, Cox KL. Electronic influenza monitoring: evaluation of body temperature to classify influenza-like illness in a syndromic surveillance system. Q Manage Health Care 2009;18:91--102.