Vaccination Program Evaluation

Vaccination programs can generate large amounts of diverse types of data, from coverage estimates to operation and management indicators. Coverage information is useful in quantifying the impact these programs have in reducing disease. Management and operations data sets can be useful diagnostics in determining which parts of the systems are performing well, and which parts can be improved when striving for greater coverage. Routine immunization is often measured as part of nationwide multi-indicator cluster surveys, which also measure socio-economic and demographic indicators. Mass vaccination campaigns (house-to-house or through fixed-post) have been used to distribute polio vaccine for decades and continue to be the best method to interrupt polio transmission. We work to provide innovative indicators and diagnostics for the varied measured built upon vaccination program data, that can be used by these programs to better estimate their impact, identify poor-performing areas, and target their efforts.

ROUTINE IMMUNIZATION

  • Multi-indicator Cluster Survey Analysis
  • Administrative Coverage Analysis

MASS VACCINATION

  • Population Immunity Analysis
  • Vaccination Coverage Analysis
  • Social Mobilization Analysis
  • Management Indicator Analysis

ROUTINE IMMUNIZATION

Multi-indicator Cluster Survey Analysis

There are multiple sources of routine coverage information available, yet there is perhaps no real gold standard measurement for coverage. Household survey data such as the Demographic and Health Surveys (DHS) and the Multiple Indicator Cluster Surveys (MICS) provide sources for routine immunization coverage estimates that differ from administrative coverage metrics. Such data can be used to characterize levels of coverage, explore household to household heterogeneity (e.g. risk factors), and explore geographical heterogeneity in coverage. We work to integrate these data sources into our analyses in order to better reflect the underlying routine immunization coverage.

Administrative Coverage Analysis​

Administrative coverage refers to registry or vaccine delivery based accounting of immunization rates. Countries report routine immunization jointly to the WHO and UNICEF who provide further coverage estimates considering other data sources; administrative data is critical to understanding global and regional immunization coverage. However, administrative data can suffer from various reporting issues, for example the incomparability of the numerator (number covered) and denominator (e.g., total population). We consider insights from administrative coverage while also critically evaluating reporting issues that may exist.

MASS VACCINATION

Population Immunity Analysis

Building and sustaining population immunity to and above the herd immunity threshold is critical to the interruption of poliovirus circulation. Using dose histories from non-polio samples collected through acute flaccid paralysis (AFP) surveillance and the historical campaign calendar, we estimate the proportion of the population that is protected against poliomyelitis. We analyze how this quantity has changed over time and the likelihood that immunity is above the herd threshold across geographical areas. We implement hierarchical temporal statistical models to generate the most robust small area estimates (district and sub-district) possible. We also assess the hypothetical impact of different future vaccine choice and/or campaign calendars.

Vaccination Coverage Analysis

We study measures of vaccination coverage: lot-quality assessment sampling (LQAS), independent monitoring, EPI surveys, and serological surveys. First, we build tracking indicators that calculate the average coverage estimates, but also extend it with estimates of measurement uncertainty and measurements of heterogeneity in the sampled population. In addition to tracking individual data sources, we also look for concordance and discordance between them as a way to identify their information content and reliability. This is an alternative to validating a data source based on disease incidence, which can be difficult or impossible in disease-free areas.

Social Mobilization Analysis

One of the main lessons learned by the Global Polio Eradication Initiative (GPEI) is the importance of social mobilization and dynamic communications strategies. We study the social indicators collected during independent monitoring of campaign performance to determine which sources of information and which decision makers are most closely related to high coverage vaccination activities. We also monitor reliability and representativeness of these data sources to ensure the information collected is of the highest quality.

Management Indicator Analysis

We study operation and management indicators, e.g., vaccination team composition, reported meeting attendance, data reporting completeness and timing, workload, supervisor behavior. Our goal is to identify associations between these multiple indicators and effective mass vaccination coverage, as well as to build diagnostics around these indicators to determine internal coherence of the data, and highlight discordances which can be helpful in evaluating the reliability of the data being collected.

We study operation and management indicators, e.g., vaccination team composition, reported meeting attendance, data reporting completeness and timing, workload, supervisor behavior. Our goal is to identify associations between these multiple indicators and effective mass vaccination, as well as to build diagnostics around these indicators to determine internal coherence of the data, and highlight discordances which can be helpful in evaluating the reliability of the data being collected.