Six years ago After School Matters, the vaunted teen-activity engine spawned by Maggie Daley's Gallery 37 art-in-a-tent idea, put itself under the microscope of a team of Northwestern University researchers. ASM—which had seen rapid growth, expanding from one program to five, and would soon be operating in 60 high schools—wanted a rigorous scientific evaluation from the professors.
Psychologist Barton Hirsch and statistician Larry Hedges took the project on, devising a million-dollar study, soliciting foundation grants, recruiting colleagues, and spending three years collecting data. Their 115-page report, After-School Programs for High School Students: An Evaluation of After School Matters, was published in June and recently posted on the Wallace Foundation website, where it landed with a great big ambiguous thud.
It turns out that a comparison of teens participating in ASM to a control group of teens yielded results so negligible the authors offered a take-your-pick pair of interpretations: a "positive perspective," focusing on the few statistically significant outcomes, and a "skeptical view," noting that effect sizes were "generally small" and—oops—that "testing a more representative sample of ASM instructors may well eliminate the few positive impacts that were found."
That's because participants were chosen randomly, but their instructors were not. To give the program its "best shot" at demonstrating impact, only the best were included.
So it's possible that ASM—which had a budget of $28 million in 2010, has been the darling of the politically savvy philanthropy crowd, and is touted as a pioneering model for programs in other cities—isn't any better than, say, a slot on the school volleyball team.
Time for a little background. Back in 1991, mayoral spouse Maggie Daley, then the mother of a couple of teenagers, and her pal Lois Weisberg, then commissioner of the Department of Cultural Affairs, noticed the city didn't have much going in the way of cultural activities for teens. After-school programs were generally geared to younger kids who needed child care until parents got home from work. As luck would have it, the mayor was looking at the time for something (anything!) to mask a glaring eyesore he'd produced in the heart of downtown: an entire empty block on State Street, cleared for a development that was looking like it would never materialize. By the time summer rolled around, Gallery 37, named for the block's designation on city zoning maps, was operating a summer art camp for teens under a cluster of big white tents across the street from Marshall Field's iconic flagship store. Classified as a job-training program (though jobs for artists are notoriously scarce), it qualified for federal funding, which made everybody happy: the teens got a stipend for learning the intricacies of mosaics or watercolors, the perpetually starving artists hired to instruct them got paid, and the mayor turned his most visible white elephant into a commendable beehive of activity.
By 1996 Gallery 37 was operating in public schools, and in 2000 its apprenticeship program, offering hands-on training under professional instructors to create "job readiness," was expanded beyond the arts. Now, under the nonprofit umbrella organization After School Matters, it consists of Gallery37, Tech37, Sports37, Words37, and Science37. It's run by a full-time staff of 75 people, operating out of the city-owned Gallery 37 building on Randolph Street and in the Cultural Center, and offers programming in parks and libraries as well as schools. This year as many as 14,000 kids from Chicago's most disadvantaged high schools will participate in ASM programs.
The NU team gathered a sample of 535 kids who'd expressed interest in one of 13 apprenticeships and randomly assigned them either to participate or be part of the control group. Using surveys, interviews, school records, and observation, they tested outcomes in four main areas: positive youth development, marketable job skills, academic performance, and problem behavior. They were met with a couple of surprises early on: though they'd been told to expect little attrition, they encountered a nearly 50 percent dropout rate among participants, and while they expected to be operating in an activity desert, almost all of the control group found other after-school options. "This changed our understanding of the experimental contrast in an important way," the researchers wrote: instead of comparing ASM to no "treatment," they would be comparing it to an alternative.