WHAT WE REALLY VALUE

Beyond Rubrics in Teaching and Assessing WritingBy BOB BROAD

UTAH STATE UNIVERSITY PRESS

Copyright © 2003 Utah State University Press
All right reserved.
ISBN: 978-0-87421-553-3

Contents

List of Figures........................................................................................viiiList of Tables.........................................................................................viiiPrologue...............................................................................................ixAcknowledgments........................................................................................xiii1 To Tell the Truth: Beyond Rubrics....................................................................12 Studying Rubric-Free Assessment at City University: Research Context and Methods.....................163 Textual Criteria: What They Really Valued, Part 1....................................................324 Contextual Criteria: What They Really Valued, Part 2.................................................735 A Model for Dynamic Criteria Mapping of Communal Writing Assessment..................................119Appendix A: Assignments for English 1 Essays...........................................................139Appendix B: Selected Sample Texts from City University.................................................142Midterm essays.........................................................................................142"Anguish"..............................................................................................142"Gramma Sally".........................................................................................144"Pops".................................................................................................146End-term portfolios....................................................................................148Portfolio 2............................................................................................148"Professional Helper" (from Portfolio 3)...............................................................156Portfolio 4............................................................................................157Appendix C: Tabulation of Votes on Sample Texts........................................................165Appendix D: Sample Interview Questions.................................................................166Appendix E: Explanation of References to City University Transcripts...................................167References.............................................................................................169Index..................................................................................................173

Chapter One

TO TELL THE TRUTH Beyond Rubrics

College writing research in the disciplinary period which began, roughly, in the mid-1960s has not told us much about exactly what it is that teachers value in student writing. Researchers who have used statistical methodologies to address this question have thrown little light on the issue.... And guidelines published by English departments-at least at places where I've taught-are even less specific. An "A" paper is one that "displays unusual competence"; hence, an "A" paper is an "A" paper. Faigley, Fragments of Rationality

Consider your favorite college or university writing program. Instructors in the program may include tenure-line faculty, adjunct faculty, graduate teaching assistants, and an administrator or two. Some are new to the program; some have been there thirty years. Several of them are trained in the field of composition and teach it by choice; others teach writing only when they can't teach literature; a few are on the writing staff mainly because it's a paying job. This diverse troupe probably delivers one or two required introductory composition courses to nearly every student who appears at your institution's door. Though they diverge considerably in their backgrounds, emphases, interests, and areas of expertise in teaching rhetoric, your program's instructors almost certainly teach a clearly established curriculum, including common readings, writing assignments, writing processes, and educational goals.

Now ask yourself about these teachers of college composition the question Lester Faigley implies in the epigraph above: What exactly do they value in their students' writing? More likely than not, your writing program's best answer will be found in a rubric or scoring guide, the "guidelines published by English departments" Faigley mentions. Hundreds of such guides for writing assessment are available in books and on the worldwide web, and many writing programs have their own. A prominent example can be found in the back of Edward M. White's Teaching and Assessing Writing (298). White's "Sample Holistic Scoring Guide" ("prepared by committees in the California State University English departments, 1988") identifies six levels of rhetorical achievement. At CSU, a student's text qualifies for the highest rating ("superior") if it meets the following five criteria:

Addresses the question fully and explores the issues thoughtfully Shows substantial depth, fullness, and complexity of thought Demonstrates clear, focused, unified, and coherent organization Is fully developed and detailed Evidences superior control of diction, syntactic variety, and transition; may have a few minor flaws

As a statement of the key rhetorical values of CSU English departments, I find this guide admirable in its clarity, simplicity, and emphasis on intellectual and rhetorical substance over surface mechanics or format concerns. Furthermore, by presenting not only levels of achievement ("incompetent" to "superior") but also the five specific evaluative criteria quoted above, it goes far beyond the tautological "A = A" formulation that Faigley protests.

But does it go far enough? The strength of the hundreds of rubrics like White's lies in what they include; their great weakness is what they leave out. They present to the world several inarguably important criteria endorsed by the local writing program administrator as the criteria by which writing should be evaluated in the relevant program. They omit any mention of the dozens of other criteria (such as "interest," "tone," or "legibility") by which any rhetorical performance is also likely to be judged. In pursuit of their normative and formative purposes, traditional rubrics achieve evaluative brevity and clarity. In so doing, they surrender their descriptive and informative potential: responsiveness, detail, and complexity in accounting for how writing is actually evaluated.

We need to critically examine such representations of our rhetorical values on the basis of what they teach-and fail to teach-students, faculty, and the public about the field of writing instruction. Theories of learning, composition, and writing assessment have evolved to the point at which the method and technology of the rubric now appear dramatically at odds with our ethical, pedagogical, and political commitments. In short, traditional rubrics and scoring guides prevent us from telling the truth about what we believe, what we teach, and what we value in composition courses and programs.

Theorists of knowledge from Nietzsche to Foucault and beyond have taught us that calls for "truth" cannot go unexplained. So I propose this working definition of truth as I use it in this book: truth means doing our epistemological best. Before we make a knowledge claim (for example, Here is how writing is valued in our writing program) that carries with it serious consequences for students, faculty, and society, we need to conduct the best inquiry we can. In their rush toward clarity, simplicity, brevity, and authority, traditional scoring guides make substantial knowledge claims based on inadequate research.

A prime assumption of my work is that a teacher of writing cannot provide an adequate account of his rhetorical values just by sitting down and reflecting on them; neither can a WPA provide an adequate account of the values of her writing program by thinking about them or even by talking about them in general terms with her writing instructors. In this book I offer what I believe is a method of evaluative inquiry better grounded both theoretically and empirically, a method that yields a more satisfactory process of exploration and a more useful representation of the values by which we teach and assess writing.

WHAT WE REALLY TEACH

As a subfield of English studies, rhetoric and composition teaches and researches what educators generally accept as the preeminent intellectual skills of the university: critical and creative thinking, as well as interpretation, revision, and negotiation of texts and of the knowledge those texts are used to create (Berlin). Done well, this work prepares our students for success in personal relationships, careers, and democratic citizenship.

Most of us in the field would therefore likely embrace as part of our mission Marcia Baxter Magolda's call to help students move toward "self-authorship," defined as "the ability to collect, interpret, and analyze information and reflect on one's own beliefs in order to form judgments" (14). Unfortunately, the undergraduate experiences of participants in Baxter Magolda's study mainly lacked conditions that would have helped them develop in such sophisticated ways. As a result, during their early postcollege careers, they often struggled and stumbled in their efforts to "become the authors of their own lives." Reflecting on what higher education did and did not offer her study's participants, Baxter Magolda explains:

They would have been better prepared for these [early interpersonal, career, and citizenship] roles, and have struggled less, had the conditions for self-authorship been created during their college experience. (xxii)

The key academic principle that helps students move toward self-authorship is, according to Baxter Magolda, "that knowledge is complex, ambiguous, and socially constructed in a context" (195, Baxter Magolda's emphasis). Theory, research, and teaching in rhetoric and composition strongly support such views of knowledge, including its social and context-specific character. However, rubrics, the most visible and ubiquitous tool of writing assessment-arguably the aspect of rhetoric/composition that impinges most powerfully and memorably on our students' lives-teach our students an exactly opposite view of knowledge, judgment, and value. At the heart of our educational and rhetorical project, rubrics are working against us.

For all its achievements and successes over the past half century (see Yancey), the field of writing assessment has no adequate method for answering one of its most urgent and important questions: What do we value in our students' writing? What we have instead are rubrics and scoring guides that "over-emphasize formal, format, or superficial-trait characteristics" of composition (Wiggins 132) and that present "generalized, synthetic representations of [rhetorical] performances ... too generic for describing, analyzing, and explaining individual performances" (Delandshere and Petrosky 21). Instead of a process of inquiry and a document that would highlight for our students the complexity, ambiguity, and context-sensitivity of rhetorical evaluation, we have presented our students with a process and document born long ago of a very different need: to make assessment quick, simple, and agreeable. In the field of writing assessment, increasing demands for truthfulness, usefulness, and meaningfulness are now at odds with the requirements of efficiency and standardization. The age of the rubric has passed.

Regarding rubrics and scoring guides everywhere, I raise questions not only about their content but also about their origins and uses. To determine what we really value in a particular writing program, we must therefore pursue several related questions:

How do we discover what we really value? How do we negotiate differences and shifts in what we value? How do we represent what we have agreed to value? and What difference do our answers to these questions make?

Compositionists willing to address these questions of inquiry, negotiation, representation, and consequences, will, I believe, find traditional rubrics and scoring guides lacking crucial ethical, pedagogical, and political qualities. This book points the way toward a new method for discovering, negotiating, and publicizing what we really value in students' writing. We can reclaim what rhetoric and composition lost half a century ago when it adopted rubrics and scoring guides as its preeminent method of representing a writing program's rhetorical values.

THE BIRTH OF RUBRICS

Modern writing assessment was born in 1961 in Princeton, New Jersey. That year, Diederich, French, and Carlton of the Educational Testing Service (ETS) published Factors in Judgments of Writing Ability (ETS Research Bulletin 61-15). Basing their work on nearly a decade of research already done (at ETS and elsewhere) on writing assessment generally and especially on inter-rater agreement, Diederich, French, and Carlton declared that

The purpose of this study was to serve as a stepping stone toward closer agreement among judges of student writing ... by revealing common causes of disagreement [among those judges]. ("Abstract")

Though the coauthors of the study emphasized that "It was not the purpose of this study to achieve a high degree of unanimity among the readers but [rather] to reveal the differences of opinion that prevail in uncontrolled grading-both in the academic community and in the educated public," still they found themselves "disturbed" by the wide variability in scoring among their fifty-three "distinguished readers": "[I]t was disturbing to find that 94% of the [300] papers received either seven, eight, or nine of the nine possible grades; that no paper received less than five different grades; and that the median correlation between readers was .31" (Abstract). The next half-century of research and practice in writing assessment was definitively charted in the single word "disturbing." Diederich, French, and Carlton had purposefully set out to investigate the geography, if you will, of rhetorical values among fifty-three "educated and intelligent" readers in six fields. Yet the truth they revealed "disturbed" them. Why? Because within the world of positivist psychometrics, the world in which ETS and other commercial testing corporations still operate, precise agreement among judges is taken as the preeminent measure of the validity of an assessment. Therefore, rather than seek to understand and carefully map out the swampy, rocky, densely forested terrain of writing assessment that they found lying before them, they quickly moved to simplify and standardize it thus:

A classification of comments was developed by taking readers at random and writing captions under which their comments could be classified until no further types of comments could be found. After many changes, 55 categories of comments were adopted and arranged under seven main headings:

1. Ideas

2. Style

3. Organization

4. Paragraphing

5. Sentence structure

6. Mechanics

7. Verbal facility (Diederich, French, and Carlton 21)

Using factor analysis-one of the "statistical methodologies" Faigley views with skepticism-the ETS researchers eventually derived from those seven main headings a list of five "factors" that seemed to capture the values of their readers:

Ideas: relevance, clarity, quantity, development, persuasiveness

Form: organization and analysis

Flavor: style, interest, sincerity

Mechanics: specific errors in punctuation, grammar, etc.

Wording: choice and arrangement of words

And thus was born what became the standard, traditional, five-point rubric, by some version of which nearly every large-scale assessment of writing since 1961 has been strictly guided.

Two things leap off every page of Factors in Judgments of Writing Ability: the analytical and imaginative genius of the researchers and their sure command of methods of statistical analysis, most especially factor analysis. What is less obvious is that, because they were firmly committed to the positivist, experimentalist paradigm that ruled their day, they let a historic opportunity slip away. Confronted with an apparent wilderness of rhetorical values, they retreated to a simplified, ordered, well-controlled representation that would keep future writing assessment efforts clean of such disturbing features as dissent, diversity, context-sensitivity, and ambiguity. Three centuries earlier, officials of the Catholic Church turned away from Galileo's telescope, so "disturbed" by what it showed them that they refused to look into it again. After showing the astronomer the Church's instruments of torture to inspire his recantation and subsequent silence, they returned eagerly to Ptolemy's comfortingly familiar millennium-old map of the heavens showing earth as the motionless center of the universe.

Within the paradigms of positivist psychometrics (Moss 1994) and experimentalist methods (North), dramatic evaluative disagreements like the ones Diederich, French, and Carlton carefully documented must necessarily register as failures. Disagreement is failure because positivism presumes a stable and independent reality (in this case "writing ability" or "writing quality") that humans try more or less successfully to "measure." The fact that "94% of the papers [in their study] received either seven, eight, or nine of the nine possible grades," was "disturbing" because if evaluators had been free of "error" and "idiosyncrasy" (Diederich's terms), they would have agreed in their judgments. The obvious psychometric solution to the problem of disagreement is to rebuild readers' evaluative frameworks so they can agree more consistently and more quickly. From the beginning this has been the precise purpose of the scoring guide and rubric, and these tools have served us well: when we use them, we can reach impressively high correlations among scorers, and we can judge students' writing with remarkable speed and relative ease.

(Continues...)

Excerpted from WHAT WE REALLY VALUEby BOB BROAD Copyright © 2003 by Utah State University Press. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.