A Word About Statistics

In his excellent book about educational statistics, Measuring Up, Daniel Koretz examines the misuse of educational statistics.  Koretz is a proponent of proper use of standardized testing, but an eloquent advocate against the common misuse of those statistics.   Koretz's book derives from his statistics course for graduate students in the school of education.  It is literate, thoughtful, and informative.  One of the most important  concepts when using educational statistics is to disaggregate educational statistics to isolate causative factors.   A related concept is that one should not compare aggregate data in ways that creates misleading conclusions.

Here is a great illustration.   A person whom I respect sort of berated me for saying (correctly) that St. Cloud district Black native English speakers were showing substantial improvement in reading proficiency in the last several years.    He said I was dead wrong, and actually St. Cloud is doing even worse that St. Paul and Minneapolis.  We went around and around on that, but the problem was that he was comparing apples and oranges leading to a completely invalid conclusion.

Here are the numbers that he must have been looking at, taken from the MDE's data site:

Yes, if you aggregate all black students in this way, it really does look as though St. Paul is doing three percentage points better than St. Cloud.  But, that overlooks that MDE lumps East African immigrants who happen to be black as "black"  together with native speaking black students, and reports the average of the two groups as "black:.   And, as Koretz points out, and really any other person with rudimentary knowledge of statistics would agree, that's statistical gobbledegook.   While these numbers are correct, St. Paul's ELL students are not primarily black, whereas there are large concentrations of black East Africans in St. Cloud and Minneapolis.

As a result, when we look at those two districts, we are comparing two completely different things in the average.  

What happens when we pull ELL students out of the averages. 

This second table shows the non ELL black proficiency rates on the MCA-III for the same school districts.   Here are the non ELL Black rates since 2013.  

Looking at data in this way is important.   It allows the public, school leaders, and teachers to get better information on how they are doing.  ELL progress cannot be measured in this way.    The progress of ELL students must be measured by the rate of progress through the multiple stage of English readiness.  We must judge our progress in this area by how long it takes for ELL's to get to the point where they are ready to function in academic English with relative proficiency.

By disaggregating in this way, we can see that St. Cloud has made great progress with black students, but it still has a long way to go.   Mixing the two groups together actually confounds your ability to understand anything.   Year over year, the percentage of ELL's may be increasing or decreasing.  If it is increasing, the proficiency rates will fall, even if you ar doing well, because you are dealing with a larger number of new English learners.  If it is decreasing, the proficiency rates may rise, simply because the average ELL student is now in the district longer. 

Proficiency rates need to be used with care.   Even when you disaggregate, one must recognize that changes may reflect all sorts of factors.   They provide you with one window on how the work of teaching is going, and that is really important, but you must consider other data and other factors as well.