High-impact tutoring: Chicago, IL
MORE ABOUT THE STRATEGY USED IN THIS CASE STUDY Math curricula and interventions, Literacy curricula and interventions
At-a-Glance
Summary
In 2011, Chicago Public Schools (CPS) piloted Becoming a Man, an evidence-based school-based group counseling and mentoring program. The intervention began showing positive impacts on socioemotional outcomes and graduation rates. However, the program had limited impact on academic outcomes. Local leaders began to look for an evidence-based academic intervention to promote improved academic outcomes among CPS students.
The University of Chicago Education Lab identified high-impact math tutoring as an effective intervention. In this approach, high school students participate in daily small-group tutoring sessions during school hours. Partnering with Match Education, which developed the model, the Education Lab implemented a high-impact tutoring pilot in 2012, followed by a two-year larger scale evaluation of the program in school years 2013-15. The pilot and evaluation demonstrated strong positive results, leading to the program’s expansion into CPS high schools. As of 2022, the tutoring model was being delivered in 24 CPS high schools.
Keys to the program’s success include building relationships between students and tutors to strengthen students’ motivation; the use of full-time site directors at each school, who ensure program quality and coordinate with each school’s faculty; sustained efforts by the Education Lab to identify, pilot, evaluate, and champion the program; strong support from then-Mayor Rahm Emanuel’s administration, which built support among school leaders; and close collaboration between CPS and Match (and later Saga Education), which ensured the program was expanded into schools that were willing and able to implement it.
Obstacles faced by this program included school leaders’ concerns about program oversight and its impact on school operations, skepticism about giving non-certified teachers significant instructional time, and the need to persuade each school individually to participate.
Results and accomplishments
2x
In a randomized controlled trial, researchers at the University of Chicago Education Lab found that students who participated in Saga Education tutoring doubled (and sometimes tripled) the amount of math students learn in a single year.
48%
Participants in Saga's tutoring program in Chicago were up to 48 percent less likely to fail math courses, and up to 26 percent less likely to fail other subjects.
$6
The University of Chicago Education Lab projects a return of $6 for every $1 invested in Saga or similar high-impact tutoring programs.
- Significant and rapid improvement to math achievement: According to two rigorous, randomized controlled trials performed by researchers at the University of Chicago Education Lab, students who participated in Saga’s tutoring program gained between one to two and a half years of math in a single year. These results were groundbreaking in the education field and disproved the notion that by high school, academic interventions would be less effective.
- Continued evidence of effectiveness leads to expansion throughout the school district: The results of the the University of Chicago Education Lab evaluations persuaded Chicago Public Schools to expand the program from a pilot in a single school in 2012 to 12 high schools in Chicago in 2013, reaching over 2,000 students. As principals and school leaders witnessed Saga’s success during the 2013-2014 school year, three more schools adopted the program in the 2014-2015 school year. As data continued to demonstrate effectiveness, buy-in among school and district leaders continued to grow. Ultimately, the school district grew the Saga program to 24 schools in 2022, and added a city-wide tutor corps in other sites.
- Success in Chicago facilitates expansion to other areas: The evidence that Saga has generated in Chicago Public Schools has enabled the organization to expand work into public schools in New York, the District of Columbia, Providence RI, Broward County Florida, and Charleston SC, significantly advancing the adoption of high-impact tutoring in schools across the country.
Overview
What was the challenge?
- Struggling schools create urgency for new investments: For years, many Chicago Public Schools experienced poor academic performance, low graduation rates, high levels of violence, and below-average employment outcomes. By 2011, there was growing recognition that new interventions were needed. In an effort to improve school performance, then-Mayor Rahm Emanuel closed some public schools and announced “turnaround” policies for others. The city also invested $18 million in strategies to improve schooling environments and reduce violence.
- Momentum generated by new programs, but additional academic supports still needed: In 2011, the city piloted a program called Becoming A Man (BAM), which aimed to reduce youth violence, among other outcomes. In an evaluation conducted by the University of Chicago Education Lab, BAM’s pilot showed improvements to social-emotional outcomes and graduation rates, and was subsequently implemented across the CPS system. But the program had limited impact on academic outcomes. Local leaders began to look for an academic intervention that could complement BAM.
- Bringing an evidence-based math tutoring program to Chicago: As local leaders sought new interventions, there was strong interest in math-focused tutoring programs, as math achievement was widely recognized as the main obstacle to many students' graduation. Researchers at the University of Chicago Education Lab identified Match Education, a charter school in Boston, as one of the few tutoring approaches backed by strong evidence (Saga Education grew out of Match in 2014). Researchers visited the program in Boston and Lawrence, MA, ultimately deciding to move forward with a pilot in Chicago in 2012.
What was the solution?
- High-impact, in-school-day math tutoring as part of a student’s education: Saga Education provides daily math tutoring sessions led by trained, full-time staff to public high school students. It is built on the belief that students will only invest effort into academic material if they feel that the tutor cares about their success.
- Thoughtfully designed tutoring sessions that encourage active learning: Tutorial periods are structured to follow Saga Education’s evidence-based model. Tutoring sessions occur during the school day, rather than before or after. Saga tutors work with a few students at a time, providing significantly more personalized instruction than a larger classroom setting. The student spends the first four minutes on a silent “do now” exercise, then the bulk of the session is spent working collaboratively with the tutor on material recently covered in math classes. The final period is a brief assessment of what the student learned, which informs the next day’s lesson.. Currently, tutors in Chicago work with 4-5 students at a time, using a blended learning model that includes the use of a virtual learning platform.
- A successful pilot followed by expansion: Saga Education first arrived in Chicago through a partnership with the University of Chicago Education Lab, which piloted the intervention at Harper High School in 2012. The pilot demonstrated strong positive results, leading the Education Lab and Saga to pursue an expansion across Chicago Public Schools.
- District-wide implementation: Saga, the University of Chicago Education Lab, and Chicago Public Schools agreed to implement the program in an initial four year contract in 12 schools (serving 600 students) in 2013-2014, and then in 15 schools (serving 1,000 students) in 2014-2015. After demonstrating positive results in continued evaluation by the University of Chicago Education Lab, Chicago Public Schools renewed and expanded its contract with Saga for an additional six years. The program currently serves 2,819 students in 31 high schools across the district.
What factors drove success?
- Consistent connection with caring adults: Saga’s program is built on the belief that students will only invest effort into academic material if they feel that the tutor cares about their success. The close relationship built between student and tutor through daily tutoring sessions motivates students to make their best effort, which often leads them to be more deeply engaged in the material and eventually experience success.
- Staffing structure ensures high quality program delivery: Saga manages their own tutors with full-time site directors who monitor the tutoring sessions, oversee tutors and provide continuous feedback on instruction. Saga’s site directors liaise with the school’s math department and grade-level teams to ensure the alignment of curricula and to share information about student progress and challenges. Saga’s leaders strongly believe that these site directors are a critical piece of the model because they ensure that the program is a “turnkey” solution that can guarantee high-quality tutoring that fits well into the school and does not require extra work for existing faculty and staff.
- An independent organization that served as an incubator, evaluator, and champion: The University of Chicago Education Lab enabled Saga’s expansion throughout Chicago possible by identifying the program, providing the resources to run the first pilot, conducting the studies that demonstrated its effectiveness, and championing the program to civic leaders across the city.
- Strong mayoral support: The administration of then-Mayor Rahm Emanuel championed the Saga program from the outset. Saga leaders believe that his support was critical because he and his administration advocated for the program with school district leaders, who then helped bring individual school leaders on board.
- Close partnership with the district enables strategic school selection: In close collaboration with Rukiya Johnson, Chicago Public Schools’ then-Executive Director of STEM and Strategic Initiatives, Saga developed criteria for initial school selection, then the district chose schools that were willing and able to implement the program effectively. Evaluation of potential schools included examining the culture of the union at the school, the staff dynamic in the math department, and the ability of schools to provide Saga with dedicated classroom space within the building.
What were the major obstacles?
- Addressing reservations from school leaders on operations and oversight: To implement Saga, schools had to rearrange how staff and students spent their day, reallocate staff hours, adjust school schedules, and shift resources. In addition to these changes, Saga required schools to allow an external party into the building and classroom, which created some trepidation for some principals. In some cases, this created a need for Saga to persuade principals that all of the changes would be worth the effort. Some teachers initially were initially skeptical as well, but soon saw Saga as a teacher-friendly intervention that put little burden on teachers and supported their ability to teach to a higher level in their own classes.
- Skepticism about non-certified teachers: In many early discussions, it was common for district leaders to question whether non-certified teachers should provide a full class period worth of instruction. Given the significant time investment of daily tutoring sessions, principals and district leaders wanted to be reassured that the instruction would be effective. The substantial training provided to incoming tutors and the evaluation of the pilot helped assuage these concerns. Even more school leaders were persuaded once they saw the program’s effects on academic performance in their schools. Reflecting school and district confidence in Saga, in 2013, the district decided to provide class credit to students enrolled in Saga tutoring.
- Persuading individual school leaders to participate in Saga: According to Saga leaders, each school made decisions autonomously and did not necessarily consult with others, and the district could not direct individual principals to bring Saga into their schools. Saga staff therefore put significant time into convincing principals one at a time and helping them work through how the program could fit into the school’s unique context. In many cases, Saga then went through the same process convincing the schools’ math departments to implement the program.
Timeline
The first Match school was founded in Boston. In 2004, under the leadership of Alan Safran, the school embeds high-impact tutoring into its school day for all of its 200 students, making it the first school that centered on the model of “high dosage” math tutoring. Safran later co-founds Saga Education in 2014 with A.J. Gutierrez, a 2006 graduate of the Match School.
Responding to deep disparities in graduation rates and rates of youth violence, researchers from the University of Chicago Education Lab carried out two randomized controlled trials (one in 2009-2010, another in 2013-2015) on a program called “Becoming A Man.” Among participants, the program showed reductions in total arrests of between 28-35%, violence-crime arrests 45-50%, and other arrests by 37-43%. These substantial results helped convince school leaders that third party programs could make big impacts in public schools.
Roland Fryer, an economist at Harvard University, reached out to Match for help launching a tutoring initiative in Houston (then the 7th largest US public school district) called the Apollo 20 Project. A comparative study on the program led by Fryer showed impressive results of three to four years of math growth in one year for thousands of 6th and 9th graders
While BAM reduced criminal activity and improved graduation rates, it had little impact on academic performance. After uncovering these findings, the University of Chicago Education Lab begins focusing its attention on finding a complementary, school-based academic program with good evidence, ultimately discovering Match
To learn more about the Match program, researchers from the University of Chicago Education Lab initiate a discussion with Match in Boston and travel to Lawrence, MA to see the Match program in person. The city of Lawrence had recently gone into state receivership and contracted with Match to implement its program in two of Lawrence’s 6 high schools. Students receiving math tutoring from Match showed a 52 percentage point increase in student growth percentile (SGP), representing the greatest one-year change in SGP in high-school math ever seen in Massachusetts.
Impressed by the results among students in Lawrence, MA, the University of Chicago Education Lab and Match collaborate to launch a pilot program at Harper High School in West Englewood, Chicago, serving 50 students with 3 tutors. The Education Lab designed the pilot to measure the results of the program with a randomized controlled trial. The first year is successful, the pilot expands into a two-year commitment at Harper, with outcomes evaluated by Education Lab researchers. The results are substantial, with participating students gaining between 1-2.5 years of math in a single year. After the results are published, The New York Times publishes a story about the program.
Following the pilot, Match and the University of Chicago Education Lab brief the mayor, securing his support for the program and leading him to encourage school district officials to support its expansion where possible.
In a call with U.S. Deputy Secretary of Education Jim Shelton, the University of Chicago Education Lab, CPS and Match receive approval for the district to pay for tutoring services with Title 1 funding.
Drawing from knowledge and experience of past in-school programs, school district officials, led by Rukiya Johnson, develop a list of schools well-suited for Match’s program and convene a group of principals to get them on board. In identifying schools, the district considers each school’s union culture, the math departments, space availability, and other factors. Representatives from Match join the meetings to help make the case.
With the mayor and school district on board and participating schools identified, Match secures a two-year contract with Chicago Public Schools of $2.5M per year, covering about 80 percent of Saga’s operational costs to expand into 12 Chicago Public Schools and serve 600 students in the fall of 2013.
Match uses a multi-step recruiting process to ensure that tutors demonstrate their ability to create meaningful relationships with students and provide effective instruction. The majority of tutors were recent college graduates and site directors had experience tutoring, teaching, and/or managing classrooms.
Following the positive results from the pilot in Chicago, Saga Education is founded as an organization independent from Match Education, the Boston charter school network which developed the high-impact tutoring model.
After a successful first year of operating in Chicago Public Schools, in the fall of 2014, Saga expands into three additional Chicago Public Schools. With this addition, Saga operates in 15 schools and serves hundreds more students, approximately 1,000 in total. To support the additional expenses of the expansion, Saga raised philanthropic funding from several local foundations to supplement school district funding.
Saga Education continues to grow around the country. By the end of 2019, the organization serves 3,000 students in three districts across the country (NY, DC, and Chicago). Saga has partnered with AmeriCorps, who will help recruit tutors and provides resources for their compensation in Chicago, New York, and Washington D.C. Saga’s award from AmeriCorps is the second largest of 75 first-time grantees nationwide. Today, Saga Education serve 7,000 directly in six cities: Chicago, New York, Charleston, Broward County, Providence, and Washington D.C. The program operates in 58 high schools nationwide.
In its announcement of the distribution of the American Rescue Plan’s Elementary and Secondary School Emergency Relief (ESSER) funds, the U.S. Department of Education (USDOE) encourages districts to use American Rescue Plan ESSER Funds for high-impact in-school tutoring. The USDOE also included key design elements of the Saga program in its COVID-19 guidelines to schools and districts.
Implementation process
How did leaders confront the problem?
- A growing sense of urgency to address public school underperformance, led by the mayor: Many Chicago Public Schools in neighborhoods with high rates of poverty had seen years of poor academic performance and high rates of violence. Then-mayor Rahm Emanuel approached the University of Chicago Education Lab to discuss developing a program to address youth violence.
- Chicago deploys and evaluates a youth counseling program that emphasizes social-emotional learning: “Becoming a Man” is a social-emotional program run by the organization Youth Guidance which aims to reduce youth arrests. In an effort to help the city address high rates of violence and criminal justice involvement among disadvantaged youth, the University of Chicago Education Lab conducted an evaluation of the program. The evaluation found that the program caused significant reductions in arrests for both violent and nonviolent crime and positively impacted high school graduation rates.
- Education Lab seeks an evidence-based tutoring program: While BAM had positive impacts on graduation rates, its primary purpose was to improve social-emotional outcomes and reduce involvement in crime. The University of Chicago Education Lab researchers were interested in finding a program that focused on academic outcomes which could complement BAM, eventually discovering the Match Charter School in Boston. With Match's high-impact tutoring model showing promising results, Education Lab researchers visited the school in Boston and in Lawrence, MA. Impressed by their visit, the Education Labs invited Match to pilot their program in Chicago.
- Pilot demonstrates strong outcomes, presenting opportunities for scaling: The first implementation of Match’s model at Harper High School in Chicago showed promising results for participants. This led Match and the University of Chicago Education Lab to explore how the program could be expanded in the city. The Education Lab introduced Match to the Mayor’s Office and found that the Mayor was enthusiastic about the effort. The Mayor’s Office helped set up meetings with Chicago Schools’ Superintendent’s office staff and made clear that he supported the model.
- Chicago Public Schools move forward with Match: With promising results and a champion in the Mayor, principals and district officials agreed to implement Saga’s model of school-day intensive math tutoring. District and school leaders agreed that students behind on math could forgo some arts or music classroom time if it meant significant improvements in math. School and district leaders, working closely with Saga and the University of Chicago Education Lab, began evaluating which schools could effectively implement the model.
How was the strategy designed?
- Strategic implementation site selection: Saga in Chicago would need to be implemented school-by-school. To achieve this, CPS first came up with a list of about 20 schools who would be good candidates for Match’s services based on a set of criteria determined by Saga. A briefing was held for interested schools. Then, Rukiya Johnson, the Executive Director of STEM & Strategic Initiatives for Chicago Public Schools, invested significant time into speaking with individual schools and working with principals to determine how Saga could fit into the school’s processes.
- A tutoring model that centers relationships with students over the course of the school year: Saga leaders knew that many students who struggle with math in high school ultimately fail to graduate, and that many students faced barriers related to poverty and neighborhood violence. Therefore, Saga’s model relied on trained tutors who could serve as a positive and emotionally supportive mentor who could help assure students that they are capable of succeeding. Tutors had 100 hours of training during the summer, covering topics including how to manage students, wait time, learning to check for things like the “joy factor,” the ratio of time spent with students talking vs. tutors talking, providing precise praise vs. general praise, checks for understanding, and more.
- Structured tutoring sessions aligned with class material: Because success in math requires active learning and practice, Saga centered active learning in their tutoring model that complemented in-class instruction. Tutoring follows and reinforces the material covered in recent classes and every session begins with a “do now” problem for students to solve. The remainder of the session continually requires the student to solve problems and engage in discussion with the tutor.
- Instruction during the school day up to five times per week: Unlike after-school tutoring programs, Saga tutoring requires an entire class period in its own classroom. Saga worked with schools to determine how tutoring could fit into the school day as well as its implications for teaching schedules, union scheduling, grading, and other process needs.
How was the plan implemented?
- Following pilot and buy in from district, recruiting and selecting schools: Officials from the Chicago Public School District worked closely with Saga to develop a list of criteria with which to determine the schools that were best suited to Saga’s program, starting with a single pilot school. The criteria for school selection included consideration of each school’s student body, union culture, and math departments. Following the success of the pilot, the University of Chicago Education Lab met with several foundations to present the results and raise funds for the program’s expansion. Once funding was in place, Saga expanded to 12 Chicago schools in 2013 and 15 in 2014.
- Recruiting the first crop of tutors and site directors: Saga worked diligently to find local tutors, all college graduates, and created a new and expanded training program. Tutor candidates were required to demonstrate their ability to establish rapport with students in a mock tutoring session with current students who then voted on whether or not they felt they could see themselves learning from the tutor.
- Building relationships with schools and math departments: After identifying a list of potential schools, district officials then helped Saga convene the school principals and help persuade them to adopt the program. In addition to principals, Saga spoke with the leadership of the schools’ math departments to secure their buy-in.
- Training the first group of Chicago tutors: To account for the differences between disadvantaged students in Chicago and the environment of Boston’s Match school, Saga adapted and expanded the tutor training program. Following training, Saga planned to continue to monitor these teaching practices and refine the tutors’ skills continuously throughout the year.
- Identifying the right students: Working closely with the University of Chicago Education Lab, Saga’s leadership spoke with teachers and parents to identify students who would most benefit from daily tutoring. School staff then worked with these students to schedule tutoring into their daily class schedule.
- Launching and expanding Saga in Chicago: With schools selected, tutors and site directors placed in schools, and students enrolled in tutoring sessions, implementation of Saga tutoring began in the new schools. An evaluation was built in to the expansion, with the University of Chicago’s continuing research building a body of rigorous evidence to demonstrate Saga’s impact.
How was the approach funded?
- Education Lab funding for a pilot at Harper High School: In order to test the effectiveness of Match’s high-impact tutoring model in Chicago, the University of Chicago Education Lab received grants from local philanthropies for a pilot in Harper High School, situated in a high-poverty neighborhood, West Englewood.
- School district and philanthropic funds for expansion: Once Saga had proved its efficacy through the pilot at Harper High School, the Chicago Public School district awarded Saga a four-year contract for their services (2015-2019). The district committed about $2.5M per year, covering about 80 percent of Saga’s operational costs. In order to fund operational expenses that the district funding did not cover, Saga and the University of Chicago Education Lab raised additional funding from local foundations. Ultimately, about a half dozen foundations made contributions that met the remaining costs.
- AmeriCorps funds to expand Saga’s reach: In 2019, Saga received a national AmeriCorps award that provided additional resources. These funds could either be used to reduce the school district’s commitment or expand Saga’s tutoring services into more classrooms. The district chose to devote the funds to further expansion.
How was the approach measured and refined?
- Experimenting with tutor-student ratios: In recent years, Saga has begun using e-learning resources in tandem with in-person tutoring and are currently assessing the efficacy of new models. In the traditional Saga model, tutors would work with a total of 12-14 students. In one new model under assessment, tutors have 24 students total in their caseload. In this delivery method, tutors provide live instruction to two students at a time, while two others work on a virtual learning platform right next to the tutors in the same room. On the next day, the two sets of students switch places, so that every student is receiving full-class live attention every other day. So far, Saga’s research has shown that this model has the same efficacy as the original model but allows tutors to take on twice the caseload.
- Pilot testing e-learning models: In addition to trying new tutor-student ratios, Saga is testing a model where tutors remotely work with students while a site director supervises the students in the classroom in person. This has also shown promising early results.
- Adjusting site director qualifying criteria: Initially, many of Saga’s site directors (who monitor classrooms during tutoring sessions) were hired from the tutoring pool. Saga has since moved to hiring site directors who were former teachers of 6th to 10th grade math, with some exceptions.
- Building school capacity to implement high-impact tutoring: Over the next several years, Saga plans to develop programs that will allow schools to implement their model on their own and is working with the Chicago Public Schools system to launch its own district-led tutoring program.