You may not be aware of the amount of time and resources that we dedicate to research, develop, and maintain the ACT® test to ensure it is of the highest quality. We will describe some of those key activities for those interested in getting a peek behind the curtain.
How do we determine what to measure on the ACT?
First, ACT begins test construction by developing a blueprint, which is not that dissimilar from a building blueprint. We specify which skills will be included for each subject and how they will be measured (e.g., functions, algebra, historical passage, and literary passage).
The blueprint ensures coherence and consistency in what is measured and how it is measured across all test forms. Nearly every three years, ACT surveys educators at all levels to understand what students need to know and be able to do to succeed from elementary school to college and into the workforce. These results are used to inform the test design and to ensure the test measures what is taught in the classroom.
How do we determine that ACT items are reliable, valid, and fair?
Item quality starts with quality item writers.
ACT hires experienced item writers from diverse backgrounds and requires them to successfully complete training and meet quality standards. In terms of diversity, ACT is careful to include item writers who represent the diversity of the US population with respect to race, ethnicity, gender and geographic location to ensure that the content of the items are fair, unbiased, accessible, and non-offensive to all students.
In addition to having an external review to confirm the accuracy and clarity of items, every single test item is reviewed for bias and fairness multiple times by fairness experts. ACT conducts formal external fairness reviews of test material and of each item before any are used to calculate ACT scores.
Finally, ACT pretests every item before it appears on an operational form to verify that items are functioning properly.
Fun Fact: Each ACT item has at least 16 independent reviews before it is included on the ACT test.
When items survive pretesting and multiple reviews, they then move to operational administration to go through another stage of review. When an entire test form is assembled, items and the form are reviewed again to ensure the items conform to the blueprint, meet statistical and subject matter criteria, and are balanced in terms of focus, subject, and skill.
How do we determine that ACT scores are reliable, valid, and fair?
To ensure score comparability across ACT administrations, each test form is equated so that a 22 on one test administration means the same thing as a 22 on another administration. Given that the ACT is predominately used for college admissions, the relationship between ACT scores and college success (e.g., grades, retention, graduation) is continually evaluated. And we don’t just make sure that scores are predictive of success for the total group but also for subgroups of students (e.g., males, Hispanic students, low-income students). We also examine whether (and if so, why) prediction accuracy varies by student subgroup. In particular, we want to know whether subgroups of students do better or worse in college than what their test scores would suggest.
The collection of this evidence can take months and often years. But it is necessary.
The ACT is used to make high-stakes decisions. We owe it to all of our stakeholders – students, parents, educators, and policymakers – to demonstrate that the ACT is a reliable, fair, and valid measure of college readiness.
We have only scratched the surface in terms of describing the work that is done to ensure the quality of the ACT. To learn more, you can follow this link to the ACT technical manual.
Follow ACT
