Committee Approach to Evaluating the Condition Review Report (Decision Matrix)

The Committee first assesses the magnitude of net benefit and then the certainty about the evidence. After this assessment, readiness and feasibility from a state public health program perspective are assessed. This two-step decision process is used to guide Committee recommendations. To assure clarity and transparency, the Committee assigns codes in this process, which are then used in the development of recommendations.

The Committee adheres to the following principles in developing recommendations:

• Recommendations are evidence based. There must be scientific evidence that screening leads to improved outcomes and that these benefits outweigh the harms of screening.
• The outcomes that matter most are the health benefits to the individual being screened. The overarching goal of screening is to improve the health-related quality of life of newborns.
• Recommendations take into account the readiness of state public health systems to begin comprehensive screening and the feasibility of either beginning such activities or developing the ability to do so. Readiness assesses the current ability to implement comprehensive screening. Feasibility assesses the resource needs for effective comprehensive screening, including a general estimate of costs to adopt screening for the condition under consideration.

The Committee begins the recommendation development process by assessing the key questions in the analytic framework. The Evidence-Based Review Group (ERG) final report includes a summary of evidence table for each key question. The Advisory Committee considers if the degree of confidence in the evidence for each key question is high, moderate, low, or very low confidence. For key questions with more than one outcome of interest (e.g., net benefit of treatment), it is recommended that the Advisory Committee consider degree of confidence in evidence for each outcome. Assessment of quality requires an evaluation of the entire body of evidence for each key question, with greater attention to the highest quality evidence and, if applicable, the critical outcomes of interest.

After evaluating each key question in the analytic framework, the Committee considers the potential magnitude of net benefit of screening to individual newborns and the broader public. The Committee will also consider the total body of evidence across the analytic framework, including each key question and the body of evidence available across the analytic framework to assess the magnitude of net benefit.

Magnitude of Net Benefit Category Ratings

The Committee assigns net benefit into one of three categories:

1. Negative Net Benefit - Harms outweigh benefits.
2. Zero to Small Net Benefit - Benefits and harms closely balance. Examples include:
   • Little benefit and little harm
   • High benefit and high harm
3. Significant Net Benefit - Benefits outweigh harms.

Contextual Considerations in Assessing Net Benefit of Expanding Newborn Screening

As consideration of the totality of the framework and key questions and outcomes within the analytic framework, net benefit includes consideration of evidence for total benefits relative to total harms for adding newborn screening compared with current standard of care to identify and diagnose individuals with the health condition. In the U.S., the primary focus of considering expanded newborn screening is on the individual benefit to the newborn. Costs are not considered in assessing the magnitude of net benefit. Instead, costs are considered as a component of feasibility, which is separately evaluated.

Nonetheless, as a public health intervention, the context of screening impact on the family, community, and population health, and public health and health care systems are an undeniable part of the total net benefit and harm equation. Considerations drawn from multi-criteria decision making models such as Evidence and Value: Impact on Decision-Making (EVIDEM) applied to rare diseases² offer relevance to newborn screening. These factors include:

• Need for screening - (disease severity, size of population affected, unmet needs),
• Comparative outcomes of clinical studies - (comparative effectiveness relative to clinical detection, safety and adverse effects, patient- and family-perceived health outcomes),
• Type of benefit - (preventive benefits, therapeutic benefits),
• Comparative cost consequences - (of the intervention, other medical costs, and other non-medical costs),
• Knowledge/evidence about screening - (quality/certainty of evidence, clinical guidelines/expert consensus),
• Other quality of life factors, and
• Public health and health care system impact and trade-offs - Committee members, who represent different stakeholders within the newborn screening system, may weigh these different contextual factors differently in determining the overall net benefit and value of expanding newborn screening.

The Committee's next step is to assess the certainty of the assessment of the magnitude of net benefit. This is a judgment based on the following critical appraisal questions that address the strength of evidence for each key question and the linkages of the evidence across the analytic framework:

1. Are there critical evidence gaps in any of the key questions?
2. To what extent are the results of the studies generalizable to newborns in the U.S.?
3. Do the studies that were reviewed have the appropriate research design to answer the key questions?
4. To what extent are the studies of adequate quality for each of the key questions?
5. What is the precision of the evidence for each key question?
6. How coherent are the studies for each key question?

For many of the rare conditions considered by the Committee, there are likely to be too few studies with sufficient subjects to have narrow confidence intervals. Furthermore, heterogeneity in study design is likely to preclude meta-analysis. Therefore, analysis of coherence may be relied upon more than the precision of the evidence.

Rating Levels of Certainty of Evidence Regarding Net Benefit

The Committee classifies certainty into one of three categories. Table II describes each of these certainty categories.

Assessment of Net Benefit and Certainty: Decision Matrix Part One

The Committee uses the ratings described above for Magnitude of Net Benefit (significant/substantial, small to zero, negative) and Certainty of evidence (High, Moderate, Low) to determine the evidence rating for these matrix dimensions.

Using this part of the matrix, the Advisory Committee assigns one code to rate the evidence. These codes are interpreted as follows, and are also included in the Decision Matrix for Evidence:

1. There is High certainty that adoption of screening for the targeted condition would lead to a Significant/substantial net benefit.
2. There is Moderate certainty that adoption of screening for the targeted condition would lead to a Significant/substantial net benefit.
3. There is High or Moderate certainty that adoption of screening for the targeted condition would lead to a small to Zero net benefit.
4. There is High or Moderate certainty that adoption of screening for the targeted condition would lead to a Negative net benefit.
5. There is Low certainty regarding the net benefit from screening.

Readiness assesses the degree to which state newborn screening programs are able to implement comprehensive screening within a short time if the condition were added to the RUSP. Readiness depends on the availability of a validated high-throughput approach to screening, availability of systems for training and education within public health and of health care providers and the public, processes for quality assurance, information systems to track screening, diagnostic services, treatment, and systems for long-term follow-up. Readiness is assigned into one of three categories:

• Ready - Most public health departments could implement screening within one year if resources were available.
• Developmental - Most public health departments would require one to three years to implement screening, even if resources were available. Potential barriers include:
  • Need to develop high-throughput screening
  • Equipment, supplies, or training material have been developed but require refinement before full-scale implementation could occur
  • Expansions are needed in systems for diagnostic testing, treatment, or follow-up
• Unprepared - Most public health programs would not be able to implement screening in fewer than three years, even if resources were made available.

Feasibility is assigned into one of two categories:

1. High to Moderate Feasibility - Screening program is possible within the financial constraints of most state public health departments and the cost of screening is well balanced against the other obligations of public health programs.
2. Low Feasibility - The resources for screening are not available and the cost is not balanced against the other obligations most state health departments.

Assessment of Readiness and Feasibility: Decision Matrix Two

Once each of the readiness and feasibility ratings are assigned, the Advisory Committee uses the following public health capacity matrix dimensions to assign readiness and feasibility ratings of public health department newborn screening programs.

Using this part of the matrix, the Committee assigns one code to public health capacity. These codes are interpreted as follows, and are also included in the Decision Matrix for Public Health Capacity:

1. Most state public health departments are ready to begin comprehensive screening and screening has high to moderate feasibility.
2. Most state public health departments have developmental readiness and screening has high to moderate feasibility.
3. Most state public health departments are unprepared to begin comprehensive screening and screening has high to moderate feasibility.
4. Implementation of screening for the targeted condition has low feasibility.

The full decision matrix is presented in Table V, which incorporates the matrix for evidence (magnitude of net benefit, the certainty of evidence), and the matrix for public health capacity (feasibility and readiness of newborn screening programs to implement newborn screening). The decision matrix and ratings are intended to provide guidance to facilitate development of recommendations; the ratings are not intended to be prescriptive.

The matrix code is used to facilitate the development of the recommendation and rationale statement. Those conditions coded A1 or A2 are the strongest candidates for addition to the RUSP. For some conditions, sufficient evidence may be available that population screening does not lead to sufficient net benefit to add to the RUSP (e.g., C or D). In other conditions, there may be a need for more research to better define the net benefit (e.g., B or L) or implementation projects to improve readiness or feasibility (A2, A3, A4, or A5).