Testing, Accounting and Incentives (Part II)

  Yesterday, I started writing about the the National Resource Council's report on incentives and test-based accountability (to down load click here:  INCENTIVES.)    The report is a rich treasure trove of information about the issues involved with the use of testing as an accountability device.  If we are serious about making proper use of testing for accountability, it is imperative that the tests are reliable and valid.  If they don't measure the objectives they are designed to measure, then they are useless for accountability purposes.   If they are designed to measure things that aren't worth measuring, then they are just as useless. The NRC research study looks at a variety of important issues. What does research generally tell us about how to choose the right incentives to motivate employees, or students?   Does the same incentive system work for all employees?  What impact does an incentive system have on the kind of employee who is attracted to the profession, and who does it encourage to leave the profession?   What impact does an incentive system have on the total amount of compensation that must be paid to keep employees on the job.    What are the costs and benefits of a performance incentive system?    

One of the dangers of using test scores uncritically, arises when we use a single statistical indicator to describe test results for accountability results.  Even properly constructed valid and reliable tests present significant challenges as accountability devices.   As the NRC report explains:

Incentives for educators are rarely attached directly to individual test scores; rather, they are usually attached to an indicator that combines and summarizes those scores in some way. Attaching consequences to different indicators created from the same test scores can produce dramatically different incentives. For example, an indicator constructed from average test scores or average test score gains will be sensitive to changes at all levels of achievement. In contrast, an indicator constructed from the percentage of students who meet a performance standard will be affected only by changes in the achievement of the students near the cut score defining the performance standard.

The same data taken from the same test scores yields significantly different results, depending upon the statistical measure used.    Here, in the table below are three ersatz classes of 7 students each, for illustration purposes.  Imagine that the scores provided here are the results of a math test, for which a score of 75 is deemed proficient.     (This idea of setting a precise score as the cutoff for proficiency is a device that permeates our NCLB based education environment.)    At the bottom of each column, you will find the average class score (48, 51, and 54 respectively), the median class score (40, 51, and 54 respectively) and the percentage of students proficient (29, 14 and 42 percent respectively).  

Notice that the average score of each of the three classes is within a few points of each other.  Class C has no student doing exceptionally well, but the class average score is marginally higher.   In addition, if its teacher is to be evaluated by by the proficiency rate, class C would seem to be significantly superior to the other two.   Do you agree that Class C is doing three times as good as Class B (even though the class averages are within 3 points of each other)?!  Notice that Class B has two kids on the "bubble," that is just marginally below proficiency.   If the teacher responsible for Class B raises her students scores by 10 points, her class proficiency rating will rise from 14 percent to 42 percent, and she will be rated a super star.    If the teacher of class A raises her students scores by 10 points, she will still only have 29 percent of her students rated proficient, and we'll be saying that she made no appreciable difference in class proficiency.  

	Class A	Class B	Class C
	10	30	10
	20	40	20
	40	40	40
	40	45	70
	40	65	80
	90	68	80
	95	75	80
Average	48	51	54
Median	40	45	70
% Prof	29	14	42

When our local newspaper reports test score performance, and likely when your local newspaper reports test scores, it is reporting state measures that focus heavily on a very narrow range of measurements.    Often, the results are heavily weighted towards the performance of disadvantaged students in order to highlight efforts to close the achievement gap.   The NRC report tells us:

Given the broad outcomes that are the goals for education, the necessarily limited coverage of tests, and the ways that indicators constructed from tests focus on particular types of information, it is prudent to consider designing an incentive system that uses multiple performance measures. Incentive systems in other sectors have evolved toward using increasing numbers of performance measures on the basis of their experience with the limitations of particular performance measures. Over time, organizations look for a set of performance measures that better covers the full range of desired outcomes and also monitors behavior that would merely inflate the measures without improving outcomes.

Musings on Testing and Accountability in Education

I've been reading Diane Ravitch's book on school reform, and my last two posts have focused on topics inspired by that work.    One of her later chapters has referred me to a 2011 publication of the National Research Council and the National Academy of Sciences.   The NRC publishes some of the most thoughtful material in a variety of fields, and so I put down Ravitch for a few days and picked up NRC publication Number 12521, called INCENTIVES AND TEST-BASED ACCOUNTABILITY IN EDUCATION. You can download it for free, and you should, by clicking on the link here.  

The issue of incentives and test-based accountability arises in several contexts.    One test-based accountability approach is to impose high stakes accountability on education professionals in the form of pay increases, bonuses, or job loss, based upon the results of standardized test results.   The NCR summary of  the research seeking to validate such systems suggest that incentive based compensation has had marginal impact on student performance, at best, and that such systems carelessly designed can actually harm educational quality.   Another test-based accountability system is to impose high stakes accountability on schools or school districts.   Once again, the research raises serious questions as to whether the kind of incentives that are being advocated these days is actually making things better.  

The NRC research study looks at a variety of important issues raised by test based accountability systems in some detail.  What does research generally tell us about how to choose the right incentives to motivate employees, or students?   Does the same incentive system work for all employees?  What impact does an incentive system have on the kind of employee who is attracted to the profession, and who does it encourage to leave the profession?   What impact does an incentive system have on the total amount of compensation that must be paid to keep employees on the job.   (Here there is a strong consensus, by the way, that when compensation is contingent on performance, employer's must increase the rate of pay, in order to make up for the contingent nature of compensation.  Performance pay is more costly, on the average.)   What are the costs and benefits of a performance incentive system?    

What do we know about test design and the use of standardized testing as a measure of performance?   Are the tests that we are currently using valid and reliable enough, to actually measure high quality performance?  Is it possible that there are are highly valuable educational goals that simply are not measured by standardized testing, and what price do we pay if we focus primarily on those goals that can be readily measured on machine scored tests?     All of these issues are analyzed in the NRC report based on ten years of accumulated research, and I'll summarize some of those conclusions in upcoming posts.

I've been interested in the topic of incentives and accountability for quite some time.     We live in a time when it has become increasingly popular to think of humans as strictly economic animals, motivated entirely by material things, (read money).   And in the last decade, it has become especially popular to suggest that education could be improved if we could just develop a system of economic rewards and punishments that allegedly would cause professional teachers significantly to upgrade their teaching by connecting pay, or even tenure, on the results of standardized tests.   Just as used car sales persons are motivated to sell more cars when they work on commission, the reasoning goes, so might we significantly increase productivity in education by putting teachers on commission, providing bonuses to those who raise test scores and none to those who don't.

My interest in rewards and punishments in education was stimulated when I joined the St. Cloud School Board.  In the year before my election our school board had implemented a bonus system for principals based upon their evaluations.  Unfortunately, our principals reported that the bonus system was actually demotivating.   One of our best principals actually turned her bonus down, stating that her work for children wasn't based on receiving a bonus, but rather on her dedication to public education and to children.    I began to read some research on performance rewards.  I read the work of one of the leading employee recognition experts, Dr. Bob Nelson,  who warned that employers need to give a lot more thought to what motivates their employees.  He argues that recognition, not performance pay, is the key to motivating employees.

Recognition represents the single most validated principle for driving desired behavior and performance in today’s work environments. Compared to the average company, employees in a recognition-focused company are 5 times more likely to feel valued, 11 times more likely to feel completely satisfied, 7 times more likely to stay, and 6 times more likely to invest in the company

Recognition works, but the central problem is whether recognition by small performance pay bonuses is the kind of recognition that works in public education, and the evidence to date doesn't seem to support the idea that you can improve educational performance by implementing pay for testing results.   What is often forgotten, in the performance pay debate is that, actually, performance pay isn't all that productive even in the world of business.  Nelson writes:

Most managers [wrongly] think money is the top motivator What employees really want is to be valued for a job well done by those they hold in high esteem. As Mary Kay Ash, founder of Mary Kay Cosmetics, says, "Imagine that every person is wearing a sign around their neck that says, 'Make me feel important.'" Sure, compensation is important, but most employees consider it a right, an exchange for the work they do. As management consultant Rosabeth Moss Kanter puts it, "Compensation is a right; recognition is a gift."

Results of a recent survey by the Council of Communication Management also confirm that recognition of good performance is the top motivator of employee performance. But how many managers consider "appreciating others" to be a major function of their job today? Not many.This is true even though one-third of managers report that they themselves would rather work in an organization where they could receive better recognition.People want to feel they are making a contribution at work, and for most individuals, this is a function of having the respect of peers and colleagues, having managers who tell them when they do a good job, and being involved and informed about what's going on in their department or organization.

Nelson points out that the key to motivation is to focus on what motivates the employees who are actually doing the work.   "What motivates others is often different from what motivates you," Nelson tells managers:

In the late 1940s, Lawrence Lindahl performed classic studies about what workers want from their jobs, and those studies were repeated with similar results in the early 1980s and 1990s. Managers identified good wages, job security, and promotion or growth opportunities as the primary reasons they believed their employees worked. Employees, on the other hand,reported intangibles such as appreciation for work done, feeling "in" on things, and empathetic managers as their most desired job attributes. When employees and supervisors ranked a list of motivators from one to 10 in order of their importance to workers, workers rated "appreciation for a job well done" as their top motivator; supervisors ranked it eighth. Employees ranked "feeling in on things" as being number two in importance; their managers ranked it last at number 10.To have a motivating work environment, this perception gap between managers and employees must be closed. Managers must be sure to reward the behavior they desire with recognition that is valued and meaningful to their employees — not just themselves.

The research on employee motivation convinces me that it is dangerous to make careless assumptions about rewards, incentives and accountability.    And so, I was really interested to review the NRC's report on incentives and accountability.     I'll pick up this thread in my next post.

   

School Reform Needs Student Partners

Yesterday, I wrote about the slogan  posted at the website of Nashville Prep,  a public college preparatory school. " Champions aren´t made in the gyms. Champions are made from something they have deep inside them—a desire, a dream, a vision.” —Muhammad Ali   The point I wanted to make is that an important element of the success of schools and teachers is the level of effort expended by students.   What got me going down this path, right now, is a discussion underway on a Facebook page called "Contract for Student Achievement," which mentioned Nashville Prep.  At the same time, I've begun to read "The Death and Life of the Great American School System..How Testing and Choice are Undermining Education."   Both the Ali quotation and Ravitch's book speak to a concern I've been struggling with:  whether our current reform efforts can be successful, unless we do a better job of making learners a partner in their own success.

Just about any teacher will tell you that its way easier to teach students successfully, if they are engaged with their own learning.  Students who attend, who really want to master the material learn faster than students who don't  As a teacher, I wanted students with the attitude that nothing --not indifferent teaching, not episodes of boredom, not mediocre textbooks -- nothing is going stand in the way of my ultimate goal, to learn this material so I can be successful in my future life.   When students don't have that attitude, teaching is really hard.    We are doing a better job, I believe, in public education beginning to focus on what teachers and schools must do to remove some of the obstacles to student success in the classroom.   But, I fear, we've not yet figured out how to inspire enough of our students to kindle that individual desire, dream and vision, to make them active partners in learning.   Because, the truth of the matter is that when students are behind, it takes more effort, more courage, more persistence, and a whole lot more desire, to catch up.    We know, I believe, that students who are behind need to spend more time learning.   But if that more time learning is spent mostly engaged in what seems to be test preparation to succeed on a test that seems inconsequential for the student, how many students will be inspired to learn?

Ravich's book suggests that when a charter school displays unusual success, part of that succes results because students who fight hard to get into charters are typically the students who have become inspired to learn.   That's a debate for another time, and another post.   We can agree, can we not, that inspiring students to make the connection between learning and future success, and the connection between hard work and learning, should be a critical component to any successful school reform strategy.    None of the great ideas that are sweeping through public education today -- professional learning communities, instructional leaders, data driven teaching, differentiation, longer school days and longer school years -- will realize their potential, if the targets of these reforms don't become inspired partners in their own success.

A sign that we are possibly on the wrong road in this regard is the belief by many educators that there is no point in assigning homework to "these kids" because most of them won't do the work, and so it will widen the achievement gap.  In yesterday's post, I mentioned Frederick Douglass' decision to learn to read, despite the great risk that it entailed.  He didn't have a teacher, everything he did was homework.  Just think how much more successful we could be, if today's students figured out what Frederick Douglass figured out:  that learning through hard work, is the road to personal success and freedom!

Maybe we should spend more time trying to develop a curriculum that causes students to adopt the desire, dream and vision necessary to become successful learners.   What might that curriculum entail?  That's the subject of my next post.