Study on gunshot detection tool did not look at gunshots

Questions are being raised about a University of Chicago study which cast doubt on a gunshot detection tool called ShotSpotter.

The technology “uses a network of acoustic sensors to detect loud, impulsive sounds that may be outdoor gunfire” before using machine-learning to detect whether or not it was a gunshot. 

Then, often in under a minute, police receive information on “time, location, and number of rounds detected” to boost response and investigation times, a company representative told The College Fix. The technology is used in a variety of cities.

Mayor Brandon Johnson announced in 2024 he would end the city’s contract, and he recently pointed to the UChicago Justice Project’s study as evidence he was correct.

Professor Robert Vargas and his team looked at response times and found “non-gunshot emergency response times also improved in areas that formerly had ShotSpotter sensors.”

The researchers also found “beats in Chicago that formerly had ShotSpotter sensors saw an average response time improvement of 4.2 minutes after ShotSpotter was removed.”

Mayor Johnson cited the study as proof that he made the right decision.

The city is “[i]nvesting in people, strengthening communities, improving response times, and giving officers tools that are effective, not wasting taxpayer dollars on technology that overpolices black and brown communities without improving safety,” Mayor Johnson said in his Memorial Day speech.  

But experts say the report does not validate his position, namely, because it does not look at gunshot response times, which is the purpose of ShotSpotter. The post-ShotSpotter period also looked at lower crime fall months compared to the higher crime summer months. (A seasonally adjusted analysis did show a difference in one minute of response times).

In an email to The Fix, Robert VerBruggen made the distinction that “the main purpose of ShotSpotter is to improve response to gunfire.” 

“Research shows it does that — police learn about gunshot incidents even when no one calls 911, they get to the scene faster, and they recover more evidence,” the Manhattan Institute scholar said. “The study does not address this issue and indeed removes gunshot calls from the data,” he wrote. 

However, VerBruggen noted “in responding to more gunshot calls—many of which don’t produce any physical evidence—police by definition lose time they could have spent on other priorities, and response times to other calls might suffer. That’s the focus of the UChicago study, and it’s a fair issue to raise even if it’s not the only important one.”

In regard to the Justice Project study, VerBruggen noted that Professor  Vargas from the study explicitly said it is “not causal.” 

VerBruggen said the best way to measure causation is to perform a randomized experiment. 

However, this is not often possible, so the surveyor must compare a “treatment group” against a “control group.” Then, one can get closer to establishing cause when the former most closely aligns with the latter before the treatment is applied. 

VerBruggen found the updated analysis of the response times, adjusted for seasonality, to be more accurate. The new analysis looked at “the same seasons of the year” and separated trends in areas with versus without ShotSpotter. The addition of a better control group led to a smaller difference of just over a minute in ShotSpotter versus ShotSpotter-less areas. 

The original, however, compared non-gunshot response times in the half a year before versus after ShotSpotter was removed without accounting for the seasonal nature of crime and emergency calls, “leading to their headline finding of a more than four-minute improvement.” 

“That said, the comparison is still not perfectly apples-to-apples,” VerBruggen wrote. “ShotSpotter was specifically installed in areas with higher crime… ShotSpotter beats started at more than 15-minute response times on average, while non-ShotSpotter areas started around 10 minutes. In other words, the ShotSpotter areas had much more room for improvement.” 

Overall, VerBruggen said the available evidence is “mixed at best” in regards to ShotSpotter’s efficacy. 

“The system doesn’t magically reduce crime when it’s installed; police need to have the resources to take full advantage of its capabilities and convert the new information to real benefits for public safety. Hopefully Chicago will be able to do that this time around, once it selects and installs a new gunshot-detection system.” 

ShotSpotter also criticized the report, saying it lacked a “sound basis” evaluating the technology’s efficacy.

“We welcome serious, data-driven evaluation of public safety technology, but this analysis has significant methodological limitations and does not support the broad conclusions being drawn from it,” a spokesperson wrote.

The comments raised similar concerns as those raised by VerBruggen.

Vargas did not respond to email and phone requests for comment from The Fix about criticism of his research in the past several weeks.

A separate think tank, the Chicago Crime Lab, calculated that 85 lives were saved per year from the technology, as police were able to locate bodies that otherwise may have been left for dead.

Crime Lab Director Jens Ludwig also did not respond to emails and phone calls about the Justice Project’s methodology.

The Chicago Police Department did not comment on ShotSpotter’s efficacy and whether it thinks Chicago should bring back ShotSpotter. 

The mayor’s office did not comment when The Fix asked why Johnson believes the study supports his decision to remove ShotSpotter and whether he has data outside of the Justice Project report.

MORE: School president cites study finding guns don’t lead to more crime to oppose campus carry