Venn Diagrams to educate us about deceptive standardized test scores

In the 1980s and 1990s, standardized tests incorporated a variety of question formats. I believe they were trying to be wholistic about measuring achievement in schools. Whether those of us to took the CAT, pre-SAT, SAT, and GRE liked analogies and essays, we accepted them as part of a broad measure of aptitude.

Since 2000, I think the talk has become about achievement instead of aptitude. Who cares if you can complete analogies? We need to know that you have retained specific information. This implies that there is a certain core set of knowledge we must all have and that knowing this will be sufficient to prepare every child to be a productive worker in society.

Why this quantitative measurement approach can hide deeper problems in our students is best explained with some diagrams:

The basic idea of testing:

Yellow: the test

Blue: our knowledge

Why it works: If we have a lot of knowledge, or if we happen to know a lot of the knowledge that is on the test, we do well and appear to have higher achievement.

Assumption: Higher test scores mean higher achievement.

What happens over time when schools teach to the test:

The fear is that if teachers are “teaching to the test” then the scope of knowledge in the brains of kids will narrow. The blue area is smaller than before, yet the green area – test scores – appear exactly the same. It’s possible for test scores to improve while student overall knowledge decreases.

Standardized test scores can hide fragmented knowledge:

The green area is the same; test scores appear the same but the usefulness of fragmented knowledge is lower.

In this example, the student’s own knowledge is just a collection of disconnected facts. This happens when classroom instruction emphasizes memorization and regurgitation. If a student has retained all the information in the textbook, he or she will perform equally as well on a test of recall, but would have much lower aptitude to innovate, create, question, and debate. There is an absence of critical thinking, problem solving, introspection, reflection, and engagement with this knowledge.

Example: High Schools in The Gambia (where I taught science in Peace Corps from 1999-2001) are a prime example of teaching-to-the-test (in this case the WAEC test), with emphasis on memorization. Although some of the WAEC test can detect understanding, most of the students’ knowledge in science is fragmented in practice. The environmental science facts are disconnected from the biology facts, and the chemistry facts are disconnected from their understanding of why computers and cell phones work.

Essays: A better standardized testing approach for qualitatively assessing a student’s ability to synthesize information

I believe in essays. Even in science, where there is no shortage of facts we could ask students, the ability for essays to communicate understanding is greater. A good scientist can synthesize information (facts) into an argument to support a hypothesis, or to educate a policymaker, or to educate voters on the scientific basis for voting for/against a referendum. Essays reveal a lack of fragmented knowledge far better than standard quantitative test methods. I’ve become a word cruncher because a large set of words encodes a greater depth of information about one’s underlying knowledge than a set of multiple choice questions alone. The same is true in International Development – where too much emphasis has been placed on the numbers and not on the summary of what it means for those who are affected.

Today students might have more esoteric (non-testworthy) knowledge than before:

This picture shows a more realistic view of the student brain (blue – on the left). Much of what occupies his/her brain is practical, cultural, esoteric knowledge that is totally separate from school. As more of his/her brain becomes dark blue, school simply becomes more artificial, and less relevant to them. No wonder Sir Ken Robinson says kids are bored…

One way to combat this trend is to make part of education all about the production (rather than the consumption) of cultural content. I read a lot of books in school, but wrote none. I consumed a lot from Math text books, but today if I were in school, I’d want my teacher to require me to make math animations like VI HART:

Or, alternatively, education could be the engine for producing cultural context. This is, after all, what keeps most social scientists and liberal arts professors in business – why not let the masses do it instead, and let the best contexts “win” through Darwinian principals? Man, that would make teaching evolution and natural selection so much easier, because you’d have it coming from an English teacher and a science teacher. All that fragmented knowledge… starting to come together. More about my teaching philosophy right here: bam!

Conspiracy corner:

Remember how I said that education should help students innovate, create, question, and debate? Not everybody wants students who do these things… because…

And perhaps this is a coincidence, but part of one presidential candidate’s platform seems to emphasize education that avoids challenging the mindsets that students bring to class:


Of course, I was lucky enough to learn some critical thinking skills, so this is what I did:

(1) Looked up the primary reference, so make sure this is not just political misinformation. The text of the GOP document on page 12 reads:

Knowledge-Based Education – We oppose the teaching of Higher Order Thinking Skills (HOTS) (values clarification), critical thinking skills and similar programs that are simply a relabeling of Outcome-Based Education (OBE) (mastery learning) which focus on behavior modification and have the purpose of challenging the student’s fixed beliefs and undermining parental authority.

(2) I will be importing the whole document into my word crunching tools, so I can see the totality of this platform. But before I do that, here’s another interesting tidbit:

Textbook Review – Until such time as all texts are required to be approved by the state board of education (SBOE), each ISD that uses non-SBOE approved instructional materials must verify them as factually and historically correct. Also the ISD board must hold a public hearing on such materials, protect citizen’s right of petition and require compliance with TEC and legislative intent. Local ISD boards must maintain the same standards as the SBOE.

Who decides what the reference facts and the official history will be? In Texas, politicians (and hence, the government) decide what reality is, and was, (and likely what it will be) – not scientists. The state board of education is an elected body of 15 politicians, and scientists have rarely served on it.

Coming full circle:

Perhaps this last decade’s experiment with narrowing our education in order to raise test scores has done more to make the country like Texas, than it has to make us better “high level critical thinkers.” It’s time that students demanded more from school than their politicians are willing to give them. The right to think is asserted, not granted.

3 thoughts on “Venn Diagrams to educate us about deceptive standardized test scores

  1. This was a very interesting piece, thanks Mark. I think that there isn’t much of a “conspiracy” behind the 1% thing, but I also don’t think there is a coincidence that those who are in power try to maintain the status quo. They aren’t going to encourage different thinking because… why would they? They’re in power. Any radical rethinking of how things are or should be could only result in a negative (or at best, neutral) change for them. I don’t believe it’s a conspiracy because I don’t think that they think of it that way, I think it’s more of a subconscious thing and a social evolutionary thing (those who were in power but demanded change did less well than those who demanded the status quo).

    As someone working for an organization pressing strongly for standardized testing, I think that the higher order thinking/avoiding fragmentation issue is an important one to keep in mind. I think testing is necessary, but I think that it needs to be done correctly to avoid exactly what you reference above. It isn’t really doing that (for the most part) right now.

  2. I agree, Matt. Standardized testing works when the tests are broad and test questions are unpredictable. Just the opposite applies to our No Child Left Behind testing methods — narrow scope with explicit and predictable test material each year, so that there a disincentive for teachers to depart from covering the “essential material.”

    In life, the “essential material” is rarely what we think it is.

  3. I think that the big problem that the US has with standardized testing is that there is such a wide disparity between the top and bottom (versus, say, Sweden). So for kids in the top 1/2 of our education system, we should be hitting them from all sides with as comprehensive a testing system as we can. Essays, multiple choice, short answer, etc, and we should be forcing teachers to work together to create a cohesive curriculum that provides context and a love for learning. (This is a great example of that, by the way:

    But then we have an entirely different world of education, where kids can’t read. Testing for these kids is meant to prove that their teachers are providing them the bare minimum of what they need just to get through life. The teachers aren’t good and/or aren’t motivated and/or aren’t provided the tools they need and/or are burnt out from struggling every day, the kids don’t get what they need at home, and school administration doesn’t know how to lead/has all the same problems teachers have. These are the kids that standardized testing is generally thinking of. When I was in school these tests were a joke, something we all took, scored in the 90th percentile, and moved on from. But there are kids who can’t even read the questions. It requires great teachers, great leadership, and a lot of time to design a great system, and very few of these lower tier schools have any of those things. Which is why we end up with tests that insist on a bare minimum. Almost no one believes in the NCLB system anymore, not even most testing advocates. But coming up with a system that works for everyone is really tough. It’s a really difficult problem because it’s not even like money is the answer (by itself), there’s lots of schools that do great (because of great leadership) without money, and plenty that fail huge percentages of kids while having huge budgets. So expecting the testing problem to be fixed by increasing budgets won’t work; it’s the tests that are supposed to act as the catalyst–by making sure everyone knows what is expected. But how do we use the tests to provide direction for teaching when kids are on such incredibly different parts of the path?

    This stuff is fascinating, but also frustrating beyond belief.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s