Evaluation of a Methodology for a Collaborative Multiple Source Surveillance Network for Autism Spectrum Disorders --- Autism and Developmental Disabilities Monitoring Network, 14 Sites, United States, 2002

Persons using assistive technology might not be able to fully access information in this file. For assistance, please send e-mail to: mmwrq@cdc.gov. Type 508 Accommodation and the title of the report in the subject line of e-mail.

Kim Van Naarden Braun, PhD¹
Sydney Pettygrove, PhD²
Julie Daniels, PhD³
Lisa Miller, MD⁴
Joyce Nicholas, PhD⁵
Jon Baio, EdS¹
Laura Schieve, PhD¹
Russell S. Kirby, PhD⁶
Anita Washington, MPH^1,7
Sally Brocksen, PhD^1,8
Hossein Rahbar, PhD⁹
Catherine Rice, PhD^1

1National Center on Birth Defects and Developmental Disabilities, CDC
²University of Arizona, Tucson, Arizona
³University of North Carolina at Chapel Hill, Chapel Hill, North Carolina
⁴Colorado Department of Public Health and Environment, Denver, Colorado
⁵Medical University of South Carolina, Charleston, South Carolina
⁶University of Alabama at Birmingham, Birmingham, Alabama
⁷Battelle Memorial Institute, Atlanta, Georgia
⁸Oak Ridge Institute for Science and Education, Oak Ridge, Tennessee
⁹Michigan State University, East Lansing, Michigan

Corresponding author: Kim Van Naarden Braun, PhD, Division of Birth Defects and Developmental Disabilities, National Center on Birth Defects and Developmental Disabilities, CDC, 1600 Clifton Road, N.E., MS E-86, Atlanta, GA 30333. Telephone: 404-498-3860; Fax: 404-498-3550; E-mail: kbn5@cdc.gov.

Abstract

Problem: Autism spectrum disorders (ASDs) encompass a spectrum of conditions, including autistic disorder; pervasive developmental disorders, not otherwise specified (PDD-NOS); and Asperger disorder. Impairments associated with ASDs can range from mild to severe. In 2000, in response to increasing public heath concern regarding ASDs, CDC established the Autism and Developmental Disabilities Monitoring (ADDM) Network. The primary objective of this ongoing surveillance system is to track the prevalence and characteristics of ASDs in the United States. ADDM data are useful to understand the prevalence of ASDs and have implications for improved identification, health and education service planning, and intervention for children with ASDs. Because complete, valid, timely, and representative prevalence estimates are essential to inform public health responses to ASDs, evaluating the effectiveness and efficiency of the ADDM methodology is needed to determine how well these methods meet the network's objective.

Description of System: The ADDM Network is a multiple-source, population-based, active system for monitoring ASDs and other developmental disabilities. In 2002, data were collected from 14 collaborative sites. This report describes an evaluation conducted using guidelines established by CDC for evaluating public health surveillance systems and is based on examination of the following characteristics of the ADDM Network surveillance system: simplicity, flexibility, data quality, acceptability, representativeness, sensitivity, predictive value positive (PVP), timeliness, stability, data confidentiality and security, and sources of variability.

Results and Interpretation: Using multiple sources for case ascertainment strengthens the system's representativeness, sensitivity, and flexibility, and the clinician review process aims to bolster PVP. Sensitivity and PVP are difficult to measure, but the ADDM methodology provides the best possible estimate currently available of prevalence of ASDs without conducting complete population screening and diagnostic clinical case confirmation. Although the system is dependent on the quality and availability of information in evaluation records, extensive quality control and data cleaning protocols and missing records assessments ensure the most accurate reflection of the records reviewed. Maintaining timeliness remains a challenge with this complex methodology, and continuous effort is needed to improve timeliness and simplicity without sacrificing data quality. The most difficult influences to assess are the effects of changes in diagnostic and treatment practices, service provision, and community awareness. Information sharing through education and outreach with site-specific stakeholders is the best mechanism for understanding the current climate in the community with respect to changes in service provision and public policy related to ASDs, which can affect prevalence estimates.

Public Health Actions: These evaluation results and descriptions can be used to help interpret the ADDM Network 2002 surveillance year data and can serve as a model for other public health surveillance systems, especially those designed to monitor the prevalence of complex disorders.

Introduction

Autism spectrum disorders (ASDs) encompass a spectrum of conditions, including autistic disorder; pervasive developmental disorders not otherwise specified (PDD-NOS); and Asperger disorder. Impairments associated with ASDs can range from mild to severe. ASDs are of increasing public health concern because the number of children receiving services for these conditions is growing. Despite the need to understand ASDs better, few data are available concerning the prevalence, characteristics, and trends of these conditions. In 2000, CDC established the Autism and Developmental Disabilities Monitoring (ADDM) Network to track the prevalence and characteristics of ASDs in the United States. The ADDM network is a multiple-source, active, population-based surveillance system that reviews developmental records at educational and health sources and employs a standardized case algorithm to identify ASD cases. ADDM data are useful to understand the prevalence of ASDs and can promote improved identification, health and education service planning, and intervention for children with ASDs.

Complete, valid, timely, and representative prevalence estimates are essential to inform public health responses to ASDs. Evaluation of the effectiveness and efficiency of the ADDM methodology, described in detail elsewhere (1), is necessary to understand how well the methods meet the network's objective. This report examines the ADDM Network methodology employed by 14 collaborative sites that collected data for the 2002 surveillance year and evaluates the validity and completeness of prevalence estimates and the effect of sources of variability on intersite prevalence differences. This evaluation was conducted using guidelines established by CDC for evaluating public health surveillance systems and includes examination of the following characteristics of the ADDM Network surveillance system, including simplicity, flexibility, data quality, acceptability, representativeness, predictive value positive, sensitivity, timeliness, stability, data confidentiality and security, and sources of variability (2).

Simplicity

The simplicity of a public health surveillance system refers to both its structure and ease of operation. The simplicity of an autism surveillance system is limited by the variability of ASD signs and symptoms and methods of diagnosis (3,4). Impairments associated with ASDs can range from mild to severe. More subtle features at the less severe end of the spectrum can remain undiagnosed as they are found in children with better communication skills and average to above-average intellectual functioning. Severity also can change as the child ages or in response to effective intervention. No observable physical attribute or clinical test can define case status, nor can cases be identified at a single point in time or type of data source. A diagnosis of an ASD is made on the basis of a constellation of behavioral symptoms rather than on biologic markers; therefore, surveillance case ascertainment requires standardized interpretation of behavioral evaluations from records at both education and health facilities. A broad range of diagnoses over multiple years must be reviewed to ensure complete case finding because children rarely receive a specific diagnosis of an ASD before age 2--3 years, with a more stable diagnosis by age 8 years (5--7). The ADDM Network common methodology (Figure 1) uses a record-based surveillance system dependent on access to education, health, and service agencies (e.g., public schools, state health clinics and diagnostic centers, hospitals, and other providers for children with developmental disabilities [DDs]) to identify cases and ensure unduplicated case counting. The process for case ascertainment occurs in two phases: 1) identification of potential cases through record screening and abstraction and 2) review of abstracted information by an ASD clinician reviewer to determine whether behaviors described in the child's evaluations are consistent with the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR) (8) criteria for autistic disorder, PDD-NOS (including atypical autism), or Asperger disorder (1,9).

Accurate collection and review of detailed evaluation information from multiple data sources is time consuming, and the lack of electronic records at the majority of data sources requires additional tasks (e.g., coordination with agencies, travel, record abstraction, and data entry). Time-tracking data collected systematically by all abstractors in Arizona indicated that abstractors spent an average of 55 hours to review or abstract, or both, 100 records. Survey data from six sites indicated that a single clinician review required an average of 20 minutes under the streamlined protocol (see Predictive Value Positive) and 47 minutes under the routine protocol. Quality assurance procedures implemented throughout data collection add time, effort, and complexity to the overall system. However, a detailed, labor-intensive approach might be the only way to produce accurate prevalence estimates for this complex behavior disorder.

Flexibility

The flexibility of a public health surveillance system refers to its ability to accommodate changes in information needs or operating conditions with little additional time, personnel, or allocated funds. The flexibility of the ADDM Network methodology allows the system to add new data sources, collect additional data elements, and incorporate the evolving science of developmental disabilities (e.g., new case definitions). The ADDM methodology can adapt to changes in data elements and case definitions between surveillance years; however, retrospective changes would be limited to data already collected. ADDM Network methods rely on, and are limited by, the availability and quality of data in evaluation records and access to those records. ADDM Network surveillance activities have been expanded to monitor other developmental disabilities, including hearing loss, vision impairment, mental retardation and cerebral palsy simultaneously. ADDM Network data also can be linked to external datasets (e.g., state birth certificate files, birth defects surveillance and newborn screening data, and complementary instruments to track children's medication prescriptions).

Data Quality

Data quality refers to the completeness and validity of a surveillance system. The amount and quality of information available from the record of an existing evaluation varies within and across ADDM Network sites and is difficult to quantify. Variability in state and local regulations, regional practices for evaluating children, and the number of providers visited can affect the number and types of evaluations available. For example, in certain states, a single record is sufficient to obtain autism eligibility for special education, but other states (e.g., New Jersey) often use multiple multidisciplinary evaluations. A qualitative comparison indicates that both the amount and quality of relevant information in records in New Jersey were greater than those at other sites. Case ascertainment is influenced by the rate of referral of children for developmental evaluation and by the sensitivity of the evaluation in detecting and recording signs and symptoms of ASDs. The ADDM Network methodology maximizes data quality by evaluating the completeness of record review, maintaining reliability in data collection and coding, and cleaning the data fields. Although these measures are taken to ensure the accuracy of data capture, the validity of the conclusions is dependent on the data in the evaluation records reviewed by project staff.

Eligible records identified by data sources but not located or available for access (e.g., located at a nonparticipating school) were classified as missing. The nature of missing records might have been systematic across multiple data sources within each ADDM site, but missing records probably were nonsystematic within an individual data source. A sensitivity analysis was conducted to evaluate the effect of missing records on prevalence (see Sensitivity).

The reliability of data collection and coding was measured against standards to ensure effective initial training, identify ongoing training needs, and adhere to the prescribed methodology. These efforts support the reliability of ADDM data by quantifying potential error caused by inconsistent data collection and coding procedures. Initial and ongoing quality control reliability methods follow a set protocol (Figures 2 and 3).

The ADDM Network implements regular, extensive, and systematic data cleaning to identify inconsistencies in reviewed and abstracted data and resolve conflicts that arise. Missing race and ethnicity information was obtained through linkage with state vital birth records.

Acceptability

The acceptability of a surveillance system is demonstrated by the willingness of persons and organizations to participate in surveillance system activities. The project's overall success was dependent on acceptance of the ADDM Network by health and education sources of each site, as these sources were needed to identify cases of ASDs. Voluntary agreements (e.g., memoranda of understanding or contracts) were established between ADDM Network sites and health and education sources that authorized site personnel to review and collect information from health or education records (Table 1). ASDs were reportable conditions at three sites (Colorado, Utah, and West Virginia), giving these sites public health authority to review and collect data from health-care facilities with no separate agreements required. At six sites (Arkansas, Maryland, North Carolina, South Carolina, Utah, and West Virginia), all targeted health sources participated. At eight sites (Alabama, Arizona, Colorado, Georgia, Missouri, New Jersey, Pennsylvania, and Wisconsin), at least one targeted health facility did not participate. The project's acceptability was lower among education sources; four sites were unable to gain access to education facilities or had minimal access (Alabama, Missouri, Pennsylvania, and Wisconsin). At six sites (Arizona, Arkansas, Colorado, Maryland, New Jersey, and North Carolina), certain schools or entire districts in their surveillance area elected not to participate. In four sites (Georgia, South Carolina, Utah, and West Virginia), school participation was complete. Lack of participation by education sources caused four sites (Arizona, Colorado, New Jersey, and North Carolina) to redefine their surveillance areas after data collection had started. Project coordinators were surveyed to determine their perception of the factors that influenced acceptability by health and education sources. The most common factors reported were privacy and confidentiality concerns of the sources, including the Health Insurance Portability and Accountability Act (HIPAA), time or resources required from the sources, and the Family Education Rights and Privacy Act (FERPA). Project staff distributed literature to parents and stakeholders at multiple forums and attended conferences to increase reporting of developmental concerns to providers, understanding of the importance of population-based surveillance of ASDs, and awareness of ASD among parents and community members.

Representativeness

Correct interpretation of surveillance data requires evaluation of the representativeness and accuracy of the surveillance system in describing the occurrence of ASDs in the population. The ADDM Network 2002 surveillance year included 14 sites that accounted collectively for 10.1% of the U.S. population aged 8 years. Because participating sites were selected through a competitive federal award process and not specifically to be representative of the entire U.S. population, ADDM Network results cannot be used as a basis for estimating the national prevalence of ASDs. Two national surveys designed as random samples of the U.S. noninstitutionalized population estimated prevalence of ASDs from parental reports of autism diagnosis among children aged 6--8 years to be 7.5 and 7.6 cases per 1,000 population, respectively (10). Although generated using a different methodology, these estimates were similar to ADDM estimates, thereby providing external validation.

The denominator is another determinant of representativeness. The 2002 surveillance year sites used data from the National Center for Health Statistics (NCHS) vintage 2004 postcensal bridged-race population estimates for July 1, 2002, to obtain counts by sex and race and ethnicity of the number of children aged 8 years (11). NCHS bridged postcensal population estimates are produced by the U.S. Census Bureau immediately after a decennial census. However, trends noted between two decennial censuses can vary substantially from trends forecast in the postcensal estimates (12). For this reason, annual postcensal estimates are updated after the subsequent decennial census, and intercensal estimates are produced. Once the 2010 census has been completed and intercensal estimates are published for 2002 and beyond, the ADDM Network will recalculate previously reported prevalence estimates to evaluate the effect of any postcensal and intercensal differences within and across sites. Using postcensal estimates rather than intercensal estimates results has been demonstrated to overestimate the prevalence of a disorder; the extent might vary by race/ethnicity (13,14). The effect of postcensal and intercensal differences might not be significant for the 2002 surveillance year but will become important as the ADDM Network collects data in subsequent surveillance years and trends are examined. No better alternative has been developed for calculating prevalence for all ADDM Network sites than NCHS data.

Predictive Value Positive

Predictive value positive (PVP) is the probability that a child whose condition is consistent with the surveillance case definition actually has the disease or condition under surveillance. A clinical diagnosis of an ASD requires intensive in-person examination of a child and often interview with the primary caregivers. Clinical confirmation of all cases identified using ADDM Network methods is resource prohibitive. The ADDM Network multiple-source, active record review methodology provides a feasible approach to population-based monitoring of ASDs. However, the ADDM methodology relies on past diagnoses, special education eligibilities, and behaviors described in children's health or education records to classify a child as having an ASD. The lack of a "gold standard" in-person standardized clinical assessment to validate these methods introduces the possibility of false-positive cases.

The validity of the ADDM Network methodology for determining case status is under assessment in a study by the Georgia ADDM Network site using clinical examinations to calculate the proportion of false-positives among confirmed ASD cases using ADDM Network methods. In 2002, the University of Miami was funded as an ADDM Network grantee to validate its ASD surveillance methods. Results from this validation project indicate that the concordance between a previously documented ASD diagnosis and the ADDM Network record review case status (97%) was greater than that of a screening with the Social Communication Questionnaire (87% at a cut-off test score of 13 points) (Marygrace Yale Kaiser, University of Miami, unpublished communication, 2006). Although not compared directly to the results of a clinical examination, these data lend support to reasonable PVP of the ADDM case-status determination.

Across the 14 ADDM Network sites for the 2002 surveillance year, 57%--86% of children classified by the ADDM Network methodology as having confirmed cases of ASDs had a previous ASD diagnosis or special education classification of autism. Past assessments of ADDM Network methodology, together with another report of 93% (15), support the assumption that PVP for this subgroup of cases is high. A study noting a relatively high (36%) false-positive rate of diagnoses reported in education records in the United Kingdom examined a limited sample (n = 33) and was difficult to compare with the ADDM Network system (16). Conversely, across sites, 14%--43% of children confirmed in the ADDM Network system as having an ASD had not received an ASD classification previously. Suspicion of an ASD was noted for 6%--19% of these children, leaving 7%--31% with no previous mention in the records of an ASD. ADDM Network methods were designed to identify children with noted behaviors consistent with ASDs but who lacked a formal diagnosis; however, this group might have had the greatest potential for false-positive classification.

One final issue affecting the sensitivity and specificity of the ADDM Network methodology for the 2002 surveillance year is the implementation of a streamlined abstraction and review protocol for children with a previous ASD diagnosis. In an earlier evaluation of these methods, 97% of children aged 8 years who were identified with a previous ASD classification ultimately were confirmed by surveillance clinician reviewers as having ASDs (CDC, unpublished data, 1996). To improve timeliness, 12 of the 14 sites adopted a streamlined abstraction and review protocol for such children. The criteria used in determining which records qualified for streamlining varied by site, and the percentage of cases ascertained using the streamlined protocol ranged from 19% in Colorado to 68% in Georgia (see Sensitivity). Because streamlined abstraction involves limited data collection of behavioral descriptions beyond those required to determine case status, the 2002 ADDM Network sites were unable to evaluate the proportion of persons whose cases would not have been confirmed on the basis of a full review of the behavioral descriptions in the children's records. However, data from the four sites that implemented full abstraction and review for the 2000 surveillance year and streamlined abstraction and review for the 2002 surveillance year indicated that the potential effect of false-positives attributable to the streamlined protocol might have been minimal (weighted average: 6%).

PVP has been improved by selectively screening high-risk segments of the population, including children receiving special education services in public schools or children with select International Classification of Diseases, Ninth Edition (ICD-9) and DSM-IV-TR billing codes related to developmental disabilities in health sources, or both (8,17).

Sensitivity

The completeness of case ascertainment depends on the sensitivity of the methodology to ascertain children with ASDs in the population. To assess potential underascertainment, quantitative or qualitative examinations (or both) were performed to identify the effects of the number of home school and private school children with ASDs; nonparticipating or unidentified data sources; abstractor error; missing records; sites requesting additional ICD-9 and DSM-IV-TR codes; and differing streamlining criteria.

Private school or home school children whose conditions were consistent with the case definition might have been missed because site agreements with public schools did not include access to information on children in nonpublic schools. Data from a random weighted sample of U.S. children aged 4--17 years from the National Survey of Children's Health (NSCH) reported that 14.2% of children whose parents reported them as having a past diagnosis of autism were attending private schools, and 1.8% were home schooled (CDC, unpublished data, 2006). Although such children were not identified systematically by ADDM Network methods through review of public education records, a subgroup might have been identified through one or more health facilities at a given ADDM Network site.

Efforts were made to identify all sources that had evaluated children for ASDs. The project continually tracked new examiners and facilities identified from children's evaluation histories to ensure that all potential data sources were pursued. However, certain health and education facilities declined to participate or were not identified by project staff (See Acceptability). Using statistical capture-recapture techniques to estimate the effect of this issue on prevalence was considered, but the assumption of independence would have been violated, thereby invalidating that method. Therefore, a quantitative assessment could not be made of the extent to which missing sources affected surveillance estimates.

Results from ongoing quality control activities were used to evaluate the accuracy of the decision made by abstractors to review the record and final case determination assigned by clinician reviewers at each site. The range of percentage of concordance regarding the decision to abstract between the quality-control auditor and abstractor at each site ranged from 87% in Georgia to 100% in North Carolina and West Virginia. For clinician review, the percentage of concordance on final case definition ranged from 79% in Utah to 100% in New Jersey (Table 2). Although quality control results for certain sites were below the established threshold, records for all abstractors and clinician reviewers that fell below the threshold were resampled until the thresholds were met. In addition, the secondary clinician review process provided assurance that the primary clinician review results are an underestimate of true agreement on final case status. The clinician review process also serves to strengthen PVP as discordance on final case status can result in over- or underascertainment.

To evaluate the effect of missing records on prevalence, all children initially identified for screening from participating sources at each site were classified into three groups: 1) all requested records located, 2) certain requested records not located, and 3) no requested records located. The children were further subdivided into six strata by type of data source (education only, health only, or both) and specificity of ASD screening criterion (presence of an ASD-specific ICD-9 or DSM-IV-TR code or school eligibility, compared with all other school eligibility, ICD-9, and DSM-IV-TR codes). Data were analyzed assuming that within each type of source or ASD-specific stratum, children with missing records would have had the same likelihood of being identified as a confirmed ASD case child, had their records been located, as children for whom all records were available for review. These analyses indicated that the possible effect of missing records on prevalence underestimation ranged from 0.4% in Wisconsin to 20% in South Carolina (Table 2).

A standard basic list of ICD-9 and DSM-IV-TR codes was reviewed for the 2002 surveillance year. However, sites that also conducted surveillance for mental retardation (Arkansas, Georgia, North Carolina, South Carolina, and Utah); cerebral palsy (Alabama, Georgia, and Wisconsin); and both hearing loss and vision impairment (Georgia) requested additional ICD-9 codes. One site (Colorado) also requested codes identified as important because of specific coding practices in the area. The proportion of additional cases identified from these additional ICD-9 codes, assuming all records with these unique codes would contribute to case status, ranged from 0% in Arkansas to 5.0% in Wisconsin (Table 2). This suggests that the additional codes would not have increased prevalence estimates substantially.

The criteria used for determining which children qualified for streamlining varied by site. Seven sites (Arizona, Arkansas, Georgia, Maryland, Missouri, New Jersey, and Pennsylvania) elected to streamline children with a primary school eligibility category of autism or a broad-spectrum ASD diagnosis, whereas Utah based streamlining on autism eligibility but a more restrictive diagnosis of autistic disorder. Four sites (Alabama, Colorado, North Carolina, and Wisconsin) streamlined records only for children with an autistic disorder diagnosis. West Virginia and South Carolina did not implement the streamlined protocol for the 2002 surveillance year. To facilitate comparability between site prevalence estimates, given this potential variability in ascertainment from using different criteria for streamlining, the least conservative streamlining criteria were applied to all children abstracted at each site. The effect on prevalence ranged from 0 in New Jersey to 9.9% in Pennsylvania (Table 2).

The use of consistent methods for case identification across surveillance years enhances the ability of the ADDM Network methods to detect changes in ASD prevalence over time. However, a true increase in ASD population prevalence might be difficult to distinguish from an increase attributable to increases in provider awareness of ASDs, changes in service provision regulations or diagnostic and treatment patterns, or differences in the breadth and depth of behavioral information in evaluation records. For example, between the 2000 and 2002 surveillance years, the prevalence of ASDs in West Virginia increased 39%. A qualitative assessment of behavioral descriptions contained in their site's evaluations indicated that improvements were made in the quality and amount of information in evaluation records during this period which might have contributed to the increase. Beginning with the 2006 surveillance year, the ADDM Network will begin rating the quality of information in records to facilitate quantitative evaluation of changes in the quality of information contained in records and their effect on prevalence over time. Because ADDM Network prevalence estimates do not rely solely on a documented ASD diagnosis from a single source, they are less likely to be affected by trends in specific usage of ASD diagnoses as long as children with social, communication, and behavioral symptoms continue to be evaluated by health or education sources for treatment or services, or both.

Although ADDM Network methods are subject to these challenges, recent studies have demonstrated that aggregate administrative data (e.g., autism eligibility data from the U.S. Department of Education) are not optimal for measuring period prevalence or monitoring changes over time. The ADDM Network's multiple-source methodology produces prevalence estimates with greater robustness to minimize classification bias than alternative available ASD prevalence measures (18--20).

Timeliness

The timeliness of the surveillance system is the speed of progression from identifying data sources to releasing results. The ADDM Network population-based surveillance system can be resource and time intensive, particularly at its inception at a new site, as evidenced by the multitude of data sources required for participation, high volume of records for review, and abstraction and clinician review and time estimates previously reported for each step in the process (Table 1). Each site must first identify potential sources for identification of potential cases, obtain access to health and education records, hire and train staff, and ensure that reliability thresholds for abstractors and clinician reviewers are met. Although the ADDM sites participating in the 2002 surveillance year represent multiple grant cycles, the estimated time required for this surveillance year, from start of funding to reporting of results, was approximately 3--4 years. Once the surveillance system has been instituted at a site, these limitations to timeliness are greatly reduced for future surveillance years.

As ADDM Network surveillance methods have evolved, the time required to make data available has decreased. Multiple surveillance years can now be conducted concurrently, and clinician review has been restructured to increase efficiency. In addition, case yield is evaluated from specific ICD-9 and DSM-IV-TR codes to determine whether certain codes could be omitted, thereby reducing the number of records to review without decreasing prevalence estimates substantially. Data management methods also have improved, reducing the time from data collection to reporting of the results.

Stability

Stability is the reliability and availability of a surveillance system consistently over time. Stability of the ADDM Network system is promoted by the continuing technical support and coordination provided by CDC, which maintains consistency in methodology across sites. Computer and network support provided by CDC minimizes time lost through computer or other technical problems. Continuation of the ADDM Network has been assured through a new 4-year grant cycle for 2006--2010, and data collection for the 2004 and 2006 surveillance years are underway. Nevertheless, because ADDM Network methods rely on administrative data, changes in maintenance of records and classification and assessment of children with ASDs over time might affect ADDM Network stability.

Data Confidentiality and Security

Although not a formal attribute of the guidelines for evaluating public health surveillance systems, data confidentiality and security must be assured. The ADDM Network employs strict guidelines to maintain the highest level of data security and confidentiality. All staff members receive intensive training concerning confidentiality policies and sign nondisclosure agreements. The network employs enhanced protection of computer files and maintains information technology security procedures for the data collection instrument to ensure that the data remain secure and confidential, including Power On passwords, Windows 2000/XP/NT passwords, MS Access Workgroup Security, and MS Access Encryption. All backups of the ARCHE database are encrypted. Once the surveillance year is completed, deidentified data are submitted to the pooled dataset. Proposals to use the aggregate, deidentified data are reviewed by the principal investigators of the ADDM Network.

Sources of Variability Across ADDM Network Sites

The ADDM Network is a multiple-site, collaborative network using a common methodology. An important goal of the network is to make meaningful comparisons of prevalence across sites. Therefore, this evaluation assessed not only how well the population prevalence of ASDs is measured within each site but also how variations in the implementation of the common methodology affected comparison of prevalence across ADDM Network sites. Data collected previously using ADDM Network methods indicated the importance of education records in monitoring the prevalence of children with developmental disabilities (9,21,22). The primary difference between ADDM Network sites for the 2002 surveillance year was the ability to access education records as 4 sites had very limited or no access to education sources. The average prevalence for sites with access to both health and education sources was significantly higher (p<0.0001) than that of sites with access to health sources only (9).

All ADDM Network sites implemented a common methodology to obtain ASD prevalence. Variability across sites in specific aspects of the common protocol were introduced through attempts to improve timeliness and conduct surveillance of additional developmental disabilities, in addition to the uncontrollable variability in facility evaluation practices. Certain sources of variability are measurable for evaluation (Table 2). These sources of variability are not mutually exclusive and, therefore, cannot be summed to represent an adjusted range of potential prevalence estimates across ADDM Network sites. Moreover, these estimates are not a comprehensive list of all sources of overascertainment and underascertainment because multiple influences that might have had an effect on prevalence (e.g., quality of information in records or proportion of children who were not evaluated at any participating data source) were not quantifiable. Although evaluation results indicate variability across sites in the implementation of the common methodology, site-specific prevalence estimates are regarded as complete, valid, and accurate, and the results offer a reasonable method for comparing intersite prevalence characteristics.

The approach to streamlined abstraction and the review of additional ICD-9 billing codes varied slightly by site, as did the degree of missing records. Although consistency strengthens a common methodology, diagnostic and billing practices differed by data source within each site, and slight modifications to enhance the ability of a site to capture the true prevalence of ASD were expected. Although the quality of abstraction and clinician review inevitably will vary within and across sites, strict quality control protocols implemented by each site enabled them to monitor the variability in quality control and resolve problems quickly.

Conclusion

The ADDM Network is the only, active, ongoing, multiple-source surveillance system for tracking prevalence of ASDs and other developmental disabilities in the United States. Using multiple sources for case ascertainment strengthens the system's representativeness, sensitivity, and flexibility, and the clinician review process aims to bolster PVP. Although sensitivity and PVP are difficult to measure, ADDM methods provide the best estimate of the population prevalence of ASDs short of conducting complete population screening and diagnostic clinical case confirmation. Although the system depends on the quality and availability of information in evaluation records, extensive quality control and data cleaning protocols and assessment of missing records ensure the most accurate reflection of the records reviewed. Maintaining timeliness remains a challenge with this complex methodology; however, possibilities for streamlining to improve timeliness and simplicity without sacrificing data quality continue to be investigated. The effects of changes in diagnostic and treatment practices, service provision, and community awareness are the most difficult influences to assess.

Information sharing through education and outreach with site-specific stakeholders is the best mechanism for understanding the current climate in the community with respect to changes in service provision and public policy related to ASDs, which can affect prevalence estimates. This evaluation can be used to help interpret surveillance results and serve as a model for other systems, especially those that monitor the prevalence of complex disorders.

Acknowledgments

Additional contributors to this report included Thaer Baroud, MPH, University of Arkansas, Little Rock, Arkansas; Richard Ittenbach, PhD, University of Pennsylvania, Philadelphia, Pennsylvania; Lydia King, PhD, Medical University of South Carolina, Charleston, South Carolina; Lynne MacLeod, PhD, University of Utah, Salt Lake City, Utah; Andria Ratchford, MPH, Colorado Department of Public Health and Environment, Denver, Colorado; Jackie Roessler, MPH, University of Wisconsin, Madison, Wisconsin. Ongoing support was provided by Joanne Wojcik, Marshalyn Yeargin-Allsopp, MD, National Center on Birth Defects and Developmental Disabilities, CDC, Atlanta, Georgia. ADDM coordinators included Meredith Hepburn, MPH, University of Alabama at Birmingham; Mary Jo Lewno,University of Arkansas, Little Rock, Arkansas; Jennifer Ottolino, University of Arizona, Tucson, Arizona; Andria Ratchford, MPH, Colorado Department of Public Health and Environment, Denver, Colorado; Maria Kolotos, Johns Hopkins University, Baltimore, Maryland; Rob Fitzgerald, MPH, Washington University in St. Louis, St. Louis, Missouri; Laura Davis, MPH, University of North Carolina at Chapel Hill; Susie Kim, MPH, New Jersey Medical School, Newark, New Jersey; Rachel Meade, University of Pennsylvania, Philadelphia, Pennsylvania; Lydia King, PhD, Medical University of South Carolina, Charleston, South Carolina; Lynne MacLeod, PhD, University of Utah, Salt Lake City, Utah; Julie O'Malley, Marshall University, Huntingdon, West Virginia; Jackie Roessler, MPH, University of Wisconsin, Madison, Wisconsin; Pauline Thomas, MD, New Jersey Medical School, Newark, New Jersey; Anita Washington MPH, Battelle Memorial Institute and National Center on Birth Defects and Developmental Disabilities, CDC, Atlanta, Georgia; Sally Brocksen, PhD, Oak Ridge Institute on Science Research and Education, Oak Ridge, Tennessee, and National Center on Birth Defects and Developmental Disabilities, CDC, Atlanta, Georgia. Additional support was provided by data abstractors, data management/programming support staff, and participating educational and clinical programs.

References

Disclaimer All MMWR HTML versions of articles are electronic conversions from ASCII text into HTML. This conversion may have resulted in character translation or format errors in the HTML version. Users should not rely on this HTML document, but are referred to the electronic PDF version and/or the original MMWR paper copy for the official text, figures, and tables. An original paper copy of this issue can be obtained from the Superintendent of Documents, U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800. Contact GPO for current prices.