Testing the Untestable: Standardized Testing and Higher-Level Thought

The perpetual thorn in the flesh of modern education is the standardized, high-stakes test. It is deeply flawed, and yet it is necessary.

It is necessary because there are more than 20,000 high schools in the United States, and there is a need to measure and evaluate students across all of them. The admissions process for colleges and universities has a practical need for a practical way to assess students.

Standardized testing is essentially flawed, yet it remains the best option, because any other conceivable procedure would be even more deeply flawed.

The flaw - or, to be more kind, limit - in any form of standardized testing is that it is forced to choose between more accurately measuring low-level knowledge or less accurately measuring high-level knowledge. Bloom’s taxonomy may be used to roughly approximate what is meant by ‘low’ or ‘high’ level knowledge in this context.

It would be a great advantage to accurately measure higher-level thought, but it is exactly this type of thinking which is most difficult to measure.

Todd Farley, an employee at the Educational Testing Corporation (ETC), explained from an insider’s perspective the processes used to write tests and process the millions of answer sheets. ETC is the parent of Educational Testing Service (ETS), which produces a whole alphabet soup of tests: AP, SAT, GRE, PSAT, NMSQT, and others.

Farley explains that, when such tests venture away from low-level knowledge, they don’t accurately measure higher-level thought.

In the words of Joseph Farrell and Gary Lawrence, attempts at assessing higher-level knowledge don’t establish “whether a student had any real competence about the subject.” Instead, such tests merely verify “that certain ‘keywords’ or concepts occured in student responses, whether or not the student actually understood their meaning.”

Standardized tests can do very well at measuring a student’s ability to do arithmetic calculations and master basic facts in history. Such tests can well measure a student’s efforts at simple examples of English spelling and grammar - including punctuation and capitalization.

Questions about analysis, synthesis, application, and evaluation - e.g., the upper levels of Bloom’s taxonomy - are, however, intrinsically ambiguous and arguable. Such questions cannot be structured for large-scale standardized testing.

Joseph Farrell and Gary Lawrence write:

Recall the critiques addressed so far with respect to some questions, namely, that the test-taker - particularly the more informed one - is put into the position of having to read the mind of the test-maker, which, as it turns out, is not one mind, but several. The test-taker, in other words, must guess at a consensus of minds, not just at one mind. Finally, the test is “pre-tested,” or to use Farley’s term, “validated,” as the test is given to a group of people to compile individual statistics for each question. Once these statistics have been compiled, they are gathered into a descriptive manual along with much other descriptive and technical matter about the test: for example, its aims, the formulas used in computing the statistics, instructions to the prospective user on how to administer the test,” and so on.

Farley offers, as do Farrell and Lawrence, numerous examples of individual test questions. Another scholar, Banesh Hoffmann, gives further instances, e.g. a question about ‘intentionally’ and ‘intensionally’ - or various uses of the word ‘emperor.’

It is demonstrable that, in many cases, badly-formed questions favor the less knowledgeable student. The more well-informed student is aware of nuances and counterexamples which speak against the answer which the test’s author hoped to elicit.

Perhaps the best tactic would be to explicitly define standardized testing as limited to lower-level history, mathematics, and English mechanics. A thorough mastery of these would be one goal of secondary education.

A second goal of secondary education would be those higher level skills which do not lend themselves to mass testing. These critical thinking skills are best learned and practiced in extended reading assignments, in classroom discussion, and in essay-writing.

Schools should not abandon teaching these skills merely because they can’t be well tested. University admissions should be based more on the lower-level skills, because they can be more accurately measured.

College Admissions: the Quest for Objective Assessment

Throughout the second half of the twentieth century, and in the first quarter of the twenty-first century, American universities have wrestled with the question of how to assess applicants. Of the many variables and many data points on an application, which most effectively reveal a student’s achievements and aptitudes? And how are they to be interpreted?

Admissions departments were faced with large increases in enrollment after WW2. This necessitated new ways of processing applications.

The numbers of applications and enrollments continued to increase throughout the remainder of the century.

New admissions processes were often less personal. A century earlier, great weight was laid on letters of recommendations. Teachers and other adults wrote letters describing not only a student’s academic successes and aptitudes, but also personal character.

In the new era of mass processing, grades on transcripts, and results from standardized tests, came into being and took on greater significance.

The hope and intent was for a more objective measurement. To some extent, that hope was fulfilled.

But, as in the case of every system, clever eyes soon began to find loopholes. Standardized tests could be taken over and over again to get better results. Special “prep courses” gave students test-taking strategies which had less to do with academic knowledge and more to do with the structure of the test questions. Tests were offered on a mix-and-match basis, whereby a student could take the best results, on each part of a test, from different administrations of the test and paste them together to make a single result.

Grades on high school transcripts could also be gerrymandered. Students enrolled under “Section 504” plans could be given modified or accommodated assignments (e.g., extra time on tests, or the opportunity to re-take quizzes on which low scores had been earned). In more than a few instances, however, this was not noted on transcripts, and so admissions officers were left to assume that the grades had been normally earned.

Likewise, essays submitted with applications are supposed to the original and unaided work of the student, yet there is a booming business among coaches who work with students to craft such writing samples.

There is, then, an endless dance, move followed by countermove, as admissions officers seek objective measurements of students, and as students and their parents work to manipulate whichever measuring system they encounter.

In Praise of Books

Literacy is the core of education. The ability to interact with text is at the heart of learning, whether the subject matter is mathematics, natural science, foreign language, or history.

The nature of text is that it is public and fixed. Any two different people can look at a book, and they will bring different perspectives to the text and form different interpretations of it: but the letters on the page are exactly the same for both readers.

But text, and thereby education itself, is being undermined by simplest maneuver: withhold books from students.

By keeping books away from students - or students away from books - , those who wish to damage education can turn text into a subjective, relativized, and fluid experience instead of a publicly accessible artifact. Tom DeWeese reports:

In New York City, administrators at the Life Sciences Secondary School have ordered all textbooks rounded up and removed. Books, they say, are antiquated. Instead, technology is to be the new god of learning.

Physical books are being replaced by data files, which can be changed continuously. A student might read a paragraph in an online history resource; when the student wishes to read that passage again a week or two later, it’s changed, and the text now reads differently.

One of the core skills which educators seek to instill is the ability of students to support theses by means of evidence. Evidence is the citation of text. But how does one cite a constantly changing fluid text?

The books were piled up in the hallway of the school. Next stop – the trash bin. Most were in good condition, including hundreds of math, algebra, geometry and various English literature text books. Also strewn around the floor were copies of Romeo and Juliet and A Street Car named Desire.

Not only has text ceased to be fixed, so that one may refer to it, but it also has become private: your copy of Hamlet or War and Peace might be different than mine.

Mechanically printed physical texts offer a standard to which common reference can be made. We can trade our differing interpretations and views by reference to, and with reference to, the fixed public text. This creates meaningful debate and dialogue.

But if my copy of For Whom the Bell Tolls has been edited to better “suit” me, based on the cookies in my browser, and your copy has been likewise altered, then any basis for intelligible dialogue has been removed.

Federal Educational Policy Causes Chaos

Federal educational policy, whether promulgated by Democrats or Republicans, is primarily an obstacle to education. For this reason, school matters are best left to state and local governments.

An example of how federal educational policy causes only confusion is seen a series of policy changes enacted during one single academic year. The 2016/2017 school year started under one set of policies. Most schools in the United States begin in August or September.

By the beginning of November 2016, Congress had enacted a new set of policies, and local schools began to shift their planning, mid-year, to comply.

In February 2017, Congress changed its policies again, as is seen from the following White House document:

H.J. Res. 58 would nullify the final rule related to the Teacher Preparation Program Accountability System, 81 Fed. Reg. 75494 (Oct. 31, 2016), promulgated by the Department of Education. This rule establishes annual State reporting to measure the performance and quality of teacher preparation programs and tie them to program eligibility for participation in the Teacher Education Assistance for College and Higher Education grant program. The rule imposes new burdensome and costly data reporting requirements on States and institutions of higher education.

So it was that a single academic year was marked by two policy changes and divided into three segments. Chaos and confusion ensued as local administrators attempted to comply; as they began to slowly redirect their sluggish bureaucracies according to one set of new mandates, another set of directives would arrive, requiring different adjustments.

Such policy mayhem is not unusual; it an established pattern.

This is merely one of many reasons why school policy is best left to states, counties, and cities.

Whither Homework?

There’s more than one way to achieve high levels of academic success. This principle is true in different aspects of education, including homework.

It is possible to structure a secondary educational institution, usually grades nine through twelve, with a rigorous program of homework. Students have specific assignments in most subjects on most evenings, and there is an accountability mechanism in place.

Such a significant homework program can lead to mastery and achievement. Whether or not it actually will produce the desired results depends in part on other variables - other aspects of the school’s program.

It is also possible to structure a high school program which leads to high levels of achievement with very little homework. This might entail, e.g., more time in class for independent practice.

Obviously, there is a question about whether homework, in and of itself, serves a purpose beyond content mastery: whether it builds work habits, study skills, and self-discipline.

Aside from that question, it is clear that a school can reach high levels of achievement with or without homework.

While either of these options might be effective, there is a third option which is less than desirable: a school which attempts to mix these two approaches.

If a school has a consistent homework policy, students are better able to adapt to it. Consistency is a virtue.

It is best for a school to be unified on the matter, with instructors following similar patterns in this regard. This is not merely a matter of school policy, but rather of school culture.

[Andrew Smith works as a German Teacher in the Ann Arbor Public Schools, serving over the decades at Huron High School, Tappan Middle School, Slauson Middle School, Scarlett Middle School, Clague Middle School, Pioneer High School, and others.]

The Future of Education Policy

Federal educational policy is always problematic, because education is primarily, and perhaps even exclusively, the business of cities, counties, and states. The national government has no clear role in education, given the text of the ninth and tenth amendments to the Constitution.

Yet, at least since the 1979 creation of the Department of Education, and perhaps even earlier, the federal government has dealt in educational policy.

What will such policy be in the near-term future? Policy statements are often vague, and citizens must often wait for the subsequent actual implementation to understand what the policies are.

In early November 2016, President Trump’s administration offered this statement:

The American Dream remains an illusion for too many families and taxpayers. We fail too often to provide our schoolchildren with a high-quality education to prepare them to be successful adults in a knowledge economy that rewards creativity and a smart work ethic, which in turns fails to provide our workforce with relevant intellectual and career competencies necessary for advancement in the U.S. or international business community. Each time our nation forgoes the academic preparedness of one child or adult we miss a window of opportunity to invent a roadmap for greatness our founding generation envisioned 229 years ago.

While an interesting assessment of education in the United States, the paragraph above sets no clear policy direction. The following paragraph begins to hint what the Trump administration might do:

For approximately 70 million school-age students, 20 million post-secondary students, and 150 million working adults, the Trump Administration will advance policies to support learning-and-earning opportunities at the state and local levels – where the heart and soul of American education takes place. We will accomplish this goal through high-quality early childhood, magnet, STEAM or theme-based programs; expansion of choice through charters, vouchers, and teacher-driven learning models; and relief from U.S. Department of Education regulations that inhibit innovation. A Trump Administration also will make post-secondary options more affordable and accessible through technology enriched delivery models.

This statement indicates that more control might be given to local educators as the quantity of regulations from the federal government decreases, that college and university tuitions might be nudged downward, and that parents and students might experience increased levels of choice in selecting a school.

It will take several years, however, to see precisely which concrete forms these policies take. Only then can the citizens observe and measure the net impact of the Trump administration’s education policy.

Emotions in School?

Much of what passes for pedagogy, as future teachers are instructed in "schools of education" at contemporary American universities, contains a hobbling internal contradiction.

On the one hand, the importance, even the supremacy, of the "affective domain" is preached to those who would become teachers. The professors, isolated in their ivory towers, far from the daily reality of an actual school filled with actual children, stress that it is important that the pupils "feel good" about being in school.

But on the other hand, the ideal archetype of a teacher is presented as someone who either never has strong emotions, or who keeps them perfectly concealed.

Inconsistently, emotions are presented as being simultaneously the centerpiece of the school and something undesirable in a school.

Current notions of "classroom management" routinely instruct the teacher never to be angry and never to project anger. The teacher is to remain calm and simply implement consequences, disagreeable or pleasant, for behavior good or bad.

To be sure, there is something appropriate in this guidance. One does not want a perpetually angry teacher with a "hair trigger" - one who's ready to begin yelling at students for minor infractions.

Professionalism dictates that a teacher deal calmly with students, explain what they've done wrong, and show them that the consequences that they receive for these actions are both natural and logical.

If we are preparing these students for higher education, for citizenship, or for the work world, we should, however, help them to understand that there are limits to this dynamic. Which employee will long survive the habit of deliberately angering his boss?

There is, at the margins, a place for a teacher to express a healthy anger. Indeed, the failure to appropriately manifest emotion is a mark of the type of dysfunction which the modern or postmodern pedagogue claims to want to avoid in the school.

The measured and professional expression of anger is a mature response to egregious behavior. Yet it is precisely this response which teachers are instructed never to make.

An adult employee in a reasonable workplace knows that it is unwise to deliberately insult or anger his colleagues or his boss. A rational employee has some unease around his employer's anger.

A student who delights in consistently provoking his fellow pupils lacks a healthy respect for the anger of others. This lack will handicap the student in current and future endeavors.

The failure to set an emotionally healthy tone, including a measured and appropriate expression of anger, limits what can be achieved in schools. By prohibiting teachers from a mature and professional expression of anger, we keep students artificially infantile in their emotional development, and allow for a student to inflict negative behavior on other students.

The results in both academic and social progress are visible.