Program Evaluation for a
Violence Prevention Initiative

RESEARCH QUESTION

What the Evaluation Set Out to Ask

This program evaluation example follows a criminology research team that is asked to evaluate a street-outreach violence prevention initiative, launched by a city agency in 30 neighborhoods. The question is whether reported violent incidents in those neighborhoods changed after the start, compared with what happened over the same months in neighborhoods the program did not reach.

The program was not randomly assigned, and it started in neighborhoods chosen because of recent violence. Places chosen that way tend to fall back toward their usual level without any intervention, so a simple before-and-after count would flatter the program. The question is framed as the program's effect on the neighborhoods it served, a reading that holds only under assumptions the plan states and checks where it can. The plan fixes the comparison neighborhoods, the pre-program window and the outcome before any counts after the start are examined.

STUDY DESIGN AND DATA

Monthly Counts for About 90 Neighborhoods

A panel of about 90 neighborhoods, 30 with the program and 60 comparison neighborhoods that met the same recent-violence criterion but were not served because of limited capacity, with monthly counts of reported violent incidents for 36 months before and 24 months after a common start month. Each record holds the neighborhood, the month, the incident count and the resident population. Every program neighborhood starts in the same month, so the timing of adoption does not vary across neighborhoods.

METHOD AND ASSUMPTIONS

Why a Comparison Group, and What Must Hold

A raw before-and-after count in the program neighborhoods mixes the program with any citywide trend and with the fall back after a spike. The comparison neighborhoods supply the trend that would have occurred without the program, so the effect is read from the change in the program neighborhoods relative to the change in the comparison neighborhoods. This holds only if the two groups would have followed parallel trends in the log of the incident rate without the program. Counts are overdispersed and populations differ, so the primary model is negative binomial with resident population as exposure, the program effect is the program-by-period interaction (a ratio of rate ratios), and standard errors are clustered by neighborhood. The dispersion choice rests on the pre-period counts, and a Poisson model with robust standard errors is run as a check.

Parallel trends cannot be proved, so the plan reads the pre-period from an event-study plot with an estimate for each quarter before and after the start, and treats a pre-trend test as limited evidence, since such a test often has too little power to detect a divergence large enough to bias the estimate. The length of the pre-program window is varied, because the fall back after a spike depends on where the window starts. Outreach can also move violence to adjacent areas or change how much is reported, so a sensitivity analysis drops comparison neighborhoods that border program neighborhoods, and a second outcome source, such as injury records, is examined where it is available.

OUTPUT AND INTERPRETATION

How the Results Are Read

The output has a flow of neighborhoods from eligibility to the analysis sample, the event-study plot, and the program-by-period rate ratio with its 95% confidence interval. The model-based difference in expected incidents per 10,000 residents per year is shown beside it, so the size of the change is readable and not only its direction. Pre-period trends in the two groups are plotted before any estimate is discussed.

Sensitivity analyses cover the pre-program window, the Poisson check, the exclusion of bordering comparison neighborhoods and the second outcome source. The estimate is read as the program's effect on neighborhoods like the 30 served, valid only if the two groups would have moved in parallel without the program, and it is reported with its precision. It does not extend to every neighborhood in the city. Whether the program continues is a decision for the city and the research team, informed by the estimate and its limits.

REPORTING

What the Researcher Received

A written report structured to TREND, the reporting guideline for nonrandomized evaluations of behavioral and public health interventions, with statistical wording for the methods, matched to the output, for the team to check and incorporate. The evaluation question, the interpretation for policy and the conclusions remain with the research team.