Testing, Testing, Testing: Tale #1

1. About cartooning

I admit, I love my little cartoon, The Parable of the Hammer. But I would hate to be as obscure as Bezonki (although I wouldn't mind drawing/painting as well as Alvey Jones). My husband thought the moral of the story of my cartoon, "When all you have is a hammer, everything looks like a nail" was obvious. And he sort of understood the references to testing being overused. But, he said, it wasn't obvious why you couldn't use a test like the NWEA MAP test--that was built for assessing individual students--to also assess teachers. Why shouldn't teachers be assessed by their students' improvement?  And this is, in fact, such a common misunderstanding of testing that I thought maybe I should address it again. [For this explanation, assume that the MAP test does a good job of assessing students, ok? A hammer, after all, does a good job pounding in nails.]

And I started thinking about this (again) because in an mlive.com article school board member Andy Thomas said that "NWEA testing that the district has engaged in will help show if there are advantages or disadvantages to a combined class." 

So in fact Andy Thomas got stuck in the same type of thinking that my husband was stuck in--that if a test can assess individual students, it can assess something else. In my husband's case, he was thinking it could assess the teacher; in Andy's case, he was thinking that it could assess types of classrooms. 

But it can't. Here's why. Students are not randomly assigned to classrooms. If a principal thinks a teacher is competent, (s)he might get--or volunteer to take--students who are perceived as more difficult.

In addition, students are not randomly assigned to schools. [Andy was talking about comparing Ann Arbor Open to other schools, and that is obviously not random since Ann Arbor Open is a magnet school.] But even if we look only at neighborhood schools, it is pretty clear that neighborhoods like Burns Park and the area near King School are in general wealthier than the area near Pittsfield or Mitchell schools.

And last, but not least, sample size is important and in general the sample sizes are too small and not at all random. Read the Northwest Evaluation Association's own memo on the subject.

So we can debate whether we should be using the NWEA MAP test to evaluate individual student progress [and you know that I'm a NO voice for that] but let's try to use the right tools for the right jobs.