Proficiency Cut Scores Leads to Watered Down Standards

From time to time, I've been talking about the use of proficiency cut scores to evaluate students and educational programs. I've been a critic of the proficiency cut score system. A proficiency cut score system sets an arbitrary standard of "proficiency" and then rates a school, a district, a state, or a nation, based on the number of students who can score higher than that proficiency standard. I'm not critical of the proficiency cut score model because I want to water down standards and make public education look better. On the contrary, I criticize the proficiency cut score model, because it actually has the opposite impact.

The mother of all cut score systems is the NAEP, the National Assessment of Educational Progress. NAEP sets arbitrary cut scores for various indicators of progress, math, science, and reading, and then tells us how many students are at the basic and proficient level. Then periodically, they issue a report on how many kids are proficient and how many are not. Now there are a number of problems with setting an arbitrary proficiency level. It suggests that there is one level for everyone, and that anyone who is not proficient is lost to mankind, and everyone who surpasses it is ready for the future. In fact, for some students, proficiency represents a tremendous waste of talent, and for other students, proficiency represents an unattainable distant goal. But today, I want to illustrate the fundamental arbitrariness of the proficiency cut score by presenting some graphs that compare the proficiency cut score used by states in comparison to the NAEP proficiency cut score. I've taken this from a Report by The Center for Public Education.



Under No Child Left Behind, schools are evaluated on whether their students pass the State's proficiency cut score. But each state can set its own cut scores arbitrarily. The Report points out that Not one state's fourth or eighth grade cutoff score for reading proficiency was as high as NAEP's (NCES 2007) "As a matter of fact, fourth grade students in 22 states can be deemed proficient in reading on their state assessment, yet be characterized as below basic in terms of NAEP." Now who is right, the NAEP or these states? And if your answer is that surely the state with the highest cut score, then step back for a minute and think what you are saying. You are really contending that proficient means the highest score identified by the politician who wants to set the highest definition for proficiency in the Country.

That begs the question whether NAEP achievement levels are unrealistic? The Report continues:

It appears that NAEP's proficient level is a good goal but not easy to achieve. As studies have shown, not even the highest performing nations would get 100 percent of their students above NAEP's proficient level. Most would not even get half. And within the United States, many high school seniors, who by other measures are high-performing, do not score above NAEP's proficiency bar.

• However, the NAEP's proficiency bar is not specifically set with the expectation that all students would be able to clear it. Proficiency is set by teachers, other educators, and subject-matter professionals who use their knowledge and expertise to set an aspirational goal for what they believe students should know and be able to do at each grade level. In contrast, state assessments are mandated by NCLB to set the proficiency bar at the level all students are expected to clear at their grade level.

• Policymakers need to consider what are good goals for educational purposes compared to what is appropriate for accountability when establishing cut scores on their state assessments. Many experts suggest that NAEP-Basic level is a better gauge for the latter.

Just remember, how proficiency is defined is based on how proficiency is used. Considering the different purposes of NAEP and state assessments, it is not surprising each provides different results. What is most concerning is the large variance in the way that states define proficiency.

And that's exactly my point. Proficiency does not come from on high. It is not handed to us on a tablet on a mountain. Proficiency is different, depending on what you want it to mean.
Now take a look at these charts, which are linked to the Report. You can see that the states are all over the map on what proficiency means. If they set low standards, then their politicians can say that their schools are doing poorly. If they set high standards, then their politicians can boast that more of their students are proficient. And all of this governs which schools are penalized under No Child Left Behind.


Look at the spread in standards displayed here. A student in North Carolina is "proficient" in reading at grade 4 when she scores 180. A student in nearby South Carolina is proficient when she scores 230. Does it tell us anything at all about North Carolina and South Carolina to report that more students in North Carolina are proficient than students in South Carolina at the same grade level? Why should the schools in South Carolina be subjected to no child left behind penalties if their students don't make adequate yearly progress towards all students scoring 230 (which by the way won't likely happen in the next 50 years), whereas the schools in North Carolina are scored under NCLB if their students all score at the NAEP "below basic" level.
More to come.