Principal Investigator: Deborah Lowe Vandell
Researchers: Valerie Hall & Pilar O’Cadiz
Funding: U.S. Department of Education
Description
This project was a 3-year evaluation of a blended model of federally funded supplemental educational services (SES) and 21st Century Community Learning Center (CCLC) afterschool programming being implemented in a local school district, in partnership with a community based organization. The model includes both afterschool enrichment activities and an intensive, research-based instructional program aimed at improving student performance in English/language arts and math. Through the collection and analysis of survey data, program attendance data, achievement test scores, program observations, focus groups and individual interviews, the project sought to identify best practices related to the implementation of the blended model of SES and afterschool programming as well as the effectiveness of the model in recruiting and retaining students to the SES program and the impact of program participation on student achievement in English/language arts and math.
The study documented progress made by THINK Together in achieving the following four performance indicators of the HQSES project in its first, second and third year of implementation:
- The number of students who enroll in THINK Together SES.
- The number of eligible students who complete full programs of THINK Together SES.
- The percentage of enrolled students, especially the lowest-achieving students, who improve their academic performance on the CSTs in Language Arts or Math.
- The number of positive, supportive relationships that students report having with adults and peers.
The Year Three study used both quantitative and qualitative methods with measures for each of the four HQSES performance indicators, including: student program enrollment, attendance, and completion data; standardized test scores; parent post surveys; student pre and post surveys; observations of SES tutoring sessions and afterschool program activities; observations of staff development activities; focus group interviews with tutors and parents; interviews with Site Coordinators and program administrators; and examination of program curriculum, tutor support and staff development materials.
Key Findings
Key Qualitative Findings
Stakeholder Satisfaction
- In surveys and interviews parents, school principals, SES instructors, and site coordinators expressed satisfaction with the general quality of the SES and ASP programs.
- Overall, survey results show that parents were satisfied with the SES program, and believed that the program was a positive experience for their child.
- Principals and parents recognized the benefit of the HQSES blended program approach when asked how they believed their students might benefit from receiving a comprehensive set of program services combining both the ASP enrichment experiences and the SES intentional focused academic support.
Staff Development
- The development and refinement of academic support curricula and instructional strategies implemented by SES tutors and ASP staff across both programs fosters a sense of coherence of purpose and approach for staff, while providing access to quality academic enrichment for a larger number of students [beyond those eligible and selected for SES services].
- Creating opportunities for SES instructors to share effective tutoring practices during weekly Friday planning sessions builds a professional learning community that positively impacts the quality of tutoring services offered.
- Making available a range of curriculum resources and opportunities for the exchange of curricular ideas and instructional strategies aides SES instructors in delivering more engaging and effective tutoring sessions.
Program Quality
- Mean ratings of 85 structured observations of the SES and ASP using the Promising Practice Ratings System documented the prevalence of positive practice indicators in support of the HQSES project goals—supportive relations with adults; (2) supportive relations with peers; (3) student engagement in activities; (4) opportunities for cognitive growth; (5) appropriate program structure; (6) opportunities for autonomy; (7) setting chaos; (8) staff over-control; (9) mastery orientation.
- Ratings were high for both SES and Afterschool Program activity observations and improved in some areas from Year Two to Year Three.
- Overall program observations showed that SES instructors and ASP staff were able to create well-organized and appropriately-structured activities that were engaging to students, challenged them to solve problems, engage in group discussions, work collaboratively, and lead activities and master academic skills/knowledge and competencies in a range of activities and subject areas.
- Strong evidence of continued performance in building positive relationships with adults and their peers was documented across the three-year study, with the highest ratings on the PPRS 4 point scale achieved in this area: the mean scores for the Supportive Relationships with Adults in the ASP was 3.39 and in the SES program 3.94. Mean scores for Supportive Relationships with Peers in the ASP were 3.48 and 3.84 for the ASP and SES program respectively.
Key Quantitative Findings
Key Findings related to the SES Program, 2010-2011
- Attendance at THINK SES program was excellent. Of 146 students who attended, 65% attended at least 34 hours of a possible 39 hours. The average attendance was 31.85 hours.
- SES attendees reported positive program experiences related to both Staff and Activities and Relationships with Peers in the THINK SES Program.
- Statistically significant (p < .01) pretest- posttest gains were obtained for the THINK SES student on the THINK Assessment of Student Skills.
- Statistically significant improvements (p < .05) were found for THINK SES students from 2010 to 2011 for CST Math, CST ELA, CELDT Overall, and CELDT Reading standardized test scores.
- Students who attended the THINK SES program had significant gains in test scores, whereas non-attenders declined in performance from 2010 to 2011.
Key Findings Pertaining to All THINK Programs, 2010-2011
- Students who attended THINK programs had significantly greater increases in CST and CELDT scores than students in the SAUSD comparison group. Students who attended the SES 10-11 program, plus at least one THINK or Supplemental session, showed the greatest increase in CST scores.
- Students who attended more supplemental sessions posted greater changes in test scores than students with fewer supplemental sessions. Students who attended more THINK afterschool sessions posted greater changes in CST Math scores and CELDT Overall scores than students with fewer sessions. A greater number of SES sessions was associated with greater changes in CST ELA and CST Math scores.
Reports
Vandell, D. L., O’Cadiz, M.P., Hall, V. (2009). High Quality Supplemental Educational Services and Afterschool Partnerships Demonstration Project: An Evaluation Study of THINK Together Programs In the Santa Ana Unified School District—Year One. Report to Think Together, Santa Ana Unified School District, and U.S. Department of Education. Year One Report
Vandell, D. L., O’Cadiz, M.P., Hall, V. (2010). High Quality Supplemental Educational Services and Afterschool Partnerships Demonstration Project: An Evaluation Study of THINK Together Programs In the Santa Ana Unified School District—Year Two. Report to Think Together, Santa Ana Unified School District, and U.S. Department of Education. Year Two Report
Vandell, D. L., O’Cadiz, M.P., Hall, V. (2012). High Quality Supplemental Educational Services and Afterschool Partnerships Demonstration Project: An Evaluation Study of THINK Together Programs In the Santa Ana Unified School District—Year Three. Final study report to Think Together, Santa Ana Unified School District, and U.S. Department of Education. Year Three Report
Measures
Qualitative Measures
Parent surveys (pre- and post-participation) focusing on program choice and satisfaction
Interviews with parents, program administrators and staff, and school principals
Observations of program activities using Promising Practice Ratings System
Quantitative measures:
Student enrollment and attendance in the SES program
Student completion of SES sessions (maximum = 34 sessions)
Standardized Test Scores (Language Arts and Math) to measure academic improvement
Student surveys (pre- and post-participation) focusing on student outcomes
Student Outcome Measures:
Work Habits
The Work Habits measure is a 6-item child self-report utilizing a 5-point response scale (1 = not at all true, 5 = really true). It is an adaptation of the Work Habits scale of the Mock Report Card (Pierce, Hamm, & Vandell, 1999. The items were reworded for the 6-item child self-report and the response scale was modified. Cronbach’s coefficient alpha for the student report in the Study of Promising After-School Programs was .76. Walker & Arbreton (2004) found that similar student reports of work habits were positively associated with academic performance and school attendance.
Reading/English Efficacy & Math Efficacy
The Reading and Math Efficacy subscales are adapted from measures developed by Eccles and colleagues with excellent psychometric properties (Eccles et al., 1993). The adapted measure consists of four items about reading/English and four items about math. The items are related competence belief (students’ perception of how good they are at reading/English and math, their expectation for success in each subject, and their sense of efficacy about learning new material in each academic domain), as well as students’ subjective task value (i.e., how important it is for them to be good at reading/English and math) (Eccles et al. 1993). The response scale was modified from the original 7-point Likert scale to a 4-point scale (1 = not at all true, 4 = really true). Jacob et al. (2002) report Cronbach’s alphas of 0.83 for reading/literacy and .84 for math.
Peer Support
The Kids in My Class survey is a compilation of 18 items taken from three questionnaires developed by Gary Ladd and his colleagues (1996) and used in the NICHD Study of Early Child Care and Youth Development. It measures the child’s social support received from peers at school, engagement in physical and verbal bullying behaviors with school classmates, and perceived victimization. The child answers on a 5-point Likert scale (1=never, 5=always). Cronbach’s alphas were high for social support from peer items (.92) and moderate for perceived victimization (.81) and engagement in bullying behaviors (.78).
After-School Environment Scale
The After-School Environment Scale (ASES; Rosenthal and Vandell, 1996) is a measure of the psychosocial climate in after-school programs. It contains 36 items on three subscales: Emotional Support, Autonomy/Privacy, and Peer Affiliation. For this study we used a shortened version used in the Study of Promising After-School Programs. The response option is a 4-point Likert scale (1=never, 4=always). Cronbach’s alphas were .88 for Emotional Support, .77 for Autonomy/Privacy, and .83 for Peer Affiliation.