By Shannon Hayes & Jeff AllenIn these uncertain times, we can rely on data from the ACT test and other standardized assessments to better understand the pandemic’s effects on progress towards college and career readiness. Because ACT scores have the same meaning across years, the data can help us quantify lost learning opportunities and, more importantly, can lead to targeted improvement efforts.
Recently, Dr. Jeff Allen, an ACT statistician with experience analyzing ACT score trends, examined the pandemic’s effects on scores from ACT’s suite of assessments, including ACT Aspire,
PreACT, and the ACT test. His analysis of ACT test score data includes students who tested during the school day in
fall 2020, with his most comprehensive study, based on 11th grade students who tested during the school day in spring 2021, coming soon.
I synthesized Jeff’s work into a research summary,
Learning Opportunities: Understanding Scores from ACT’s Assessment Suite During the COVID-19 Pandemic, outlining findings for
schools testing similar numbers of students before and during the pandemic, with tests administered
in person. The research shows that there were pandemic-related score declines which were mostly consistent across demographic groups and which tended to be more severe for younger students. I recently interviewed Jeff to learn more about the most important takeaways from his research.
Shannon: Jeff, how do we quantify the amount of learning students may need to catch up on during summer school or when they return to school in the fall based on these score declines?
Jeff: We can express changes in average test scores in terms of
comparable instructional months. For example, for the spring 2021 ACT analysis, we found that ACT math scores declined by 0.63 score points. A score decline of this magnitude is like losing more than
three months of schooling. This suggests that students experienced growth in academic readiness during the pandemic, but not to the level that we would have expected during normal circumstances. It can also help us understand how effective interventions must be to catch up.
Shannon: How well does the data represent the national population of students or specific subgroups (for example, whether students were enrolled remotely or in-person)?
Jeff: The research I have done is based on school-day testing programs. Unlike ACT tests that take place on Saturday mornings, school-day testing is usually provided to all students at a school in a particular grade level. I only included schools that tested at least 75% of their students and tested a similar number of students before and during the pandemic. This subset includes thousands of schools across the country and gets us closer to an apples-to-apples comparison of test scores across years. Unfortunately, we do not know which students participated in remote, in-person, or hybrid forms of schooling. Because the tests were administered in person and at school, it’s likely that those who learned remotely are underrepresented in the data. If remote learning was not as effective as in-person learning, the score declines we observed are probably understating the pandemic’s overall impact.
Shannon: What do the data say about students who have been historically underserved in the educational system, including students who are Black, Latinx, Native American, and Native Hawaiian/Pacific Islander, as well as those whose families face economic challenges?
Jeff: The results vary across assessments and grade levels, but we are generally finding that the score declines are slightly less severe for traditionally underserved groups.
This does not mean that longstanding opportunity gaps closed, just that they did not grow larger. For example, for the spring 2021 ACT analysis, the decline in average ACT Composite scores was least severe for Asian students. The score decline for Black and Native American students was less severe than for White and Hispanic students, who saw about the same average score decline. Although the opportunity gap did not widen, a troubling gap was still observed: the average ACT Composite scores for Black (15.6), Hispanic (17.0), and Native American (16.1) students remained lower than that of White students (20.0). We also found that the score declines were less severe for students with lower family income (<$50,000) relative to students with higher family income. It’s also important to recognize that nationwide,
students of color were less likely to be learning in person than White students, which may have impacted their testing rates and our results.
Shannon: How many students are included in these data? Are they from every state?
Jeff: The spring 2021 ACT analysis includes
nearly 600,000 students from more than 3,900 high schools and 38 states. In 19 of those states, the majority of the 11th grade population participated in school-day ACT testing. So, it’s a very large and diverse sample.
Shannon: What would you say is the main takeaway from this research for educators?
Jeff: We know that performance on the ACT test is influenced by a student’s academic and life experiences over the course of several years. The declines in ACT scores are likely caused by several factors, including social and emotional stressors, reduced instructional time and/or quality, and decreased engagement. Because we’re all part of the education community, we all have a role to play in addressing those factors and improving the conditions for learning.
We should view ACT test scores as an indicator of progress, rather than the end goal of K-12 education. Still, the ACT score declines are cause for concern. This year’s decline in scores is more severe than any other observed in our history of school-day testing, and—all else being equal—this means that fewer students are ready for college and careers. We should also examine the data from state summative assessments, especially from grades 3-8, to understand the pandemic’s effects on younger students. If there are substantial score declines in lower grade levels, as the early research suggests is the case, we should be concerned that the pandemic’s effects on college and career readiness may persist and even grow in the coming years. It’s possible that the social and emotional stressors of the last year could have a lasting negative effect and that fewer of our younger students will be able to access a rigorous high school curriculum that traditionally improves readiness for college and careers. Time will tell, and data collected from each year’s cohort of ACT examinees will serve as a barometer of progress.
Shannon: Can you remind us where your reports can be found? And if they only have time to read one report, which would you recommend they start with?
Jeff: The reports examining the test score declines, as well as other ACT pandemic-related research, can be found
here. I would suggest starting with the
research summary you wrote, Shannon, and delving into the individual reports as needed for more details. At a later date, we will post a report detailing the spring 2021 ACT data analysis, which represents our most comprehensive study.
