Evaluation Policy

Law school courses are graded. Grades are stupid. They purport to reduce the evaluation of learning to a single number on a short, obscure scale. In order to be useful to students, evaluation has to measure more than one quantity, and must be expressed qualitatively, directed at helping the student to learn more and derive more satisfaction from learning. Grades therefore cannot possibly serve the ends of the people to whom they are “given.” On further consideration, it therefore becomes apparent that grades are “given” only in the sense that they are imposed.

Students have grades imposed upon them, and are taught to care what they are “given”: The obscenity of injustice is that the dominated are taught to kiss the rod. Evaluation should take the form of sympathetic and effective counsel: students should be mentored, else the Law School's insistence on replacing centuries of apprenticeship is a sham that reduces the quality of legal training.

But we are told that employers require grades in order to evaluate potential employees. Never mind that these supposed employers have presently no jobs to offer, having been caught with their pants entirely down amidst a permanent industry restructuring they completely failed to foresee. Never mind that we should no longer be teaching students to work for institutions such as these grade-consuming employers: instead we should be teaching students to work for clients, who are people and institutions selecting expertise rather than retaining brokers of commodity lawyer-hours, and for whom grades are no more useful in the process of selecting a lawyer than they are to the students who “earn” them. Never mind anything, in fact, that would compel us to reexamine the basic mechanisms of our practice.

Naturally, educational evaluation should involve sympathetic, critical mentoring. But that takes time. We have created a culture in which law professors declare themselves superior to the mere demands of teaching, where the customer is always either right or wrong, but never worth spending too much time on. Grades are an expression of contempt for the people who pay for the educations we are supposed to be helping them to get.

Of course the system manufactures its own consent: that's the purpose of the obscenity of injustice. Even once it is apparent that grades are stupid and their imposition is unethical, involving continuous acts of disrespect towards those we have an obligation to assist, the system refuses to accept even the possibility, let alone the imperative, of change.

My goal in these wikis is to begin the remodeling of the relationships at the heart of legal education. Not after a stack of committee reports and a long debate about how much change is consistent with the self-importance of the bloviators, but now. What my students and I are doing together will always remain far from perfect. But it refuses to surrender entirely to bullshit, which is something of a start.

Objectives

The evaluation policies in the courses that are conducted here are inherently imperfect: they continue to lead to the final absurdity of a grade. But we coexist with the grading system as a community under occupation: we cannot pretend that the system has our best interests at heart, or that it will adhere to reason. We seek to achieve our objectives for the interaction between teacher and students as a whole, creating both workflows and principles of evaluation that further those objectives:

Continuous revision

All work in these wikis, and in the courses they contain, may be revised by any student at any time. Comments and edits by other colleagues, or the fruit of reconsideration, may lead to changes, as may the experience of intervening writing. Neither the work nor the evaluation of the work ends at semester boundaries—law school is a continuous process rather than a series of discrete high-stakes resolutions.

Collaborative learning

Although students have a right to private submission of their personal essays—which the access control system of these wikis facilitates—preference is given to public submission, revision, and evaluation of collectively-edited material, thus allowing the entire class to participate in editing and learning from one another's work. Collaborative learning in wikis implies everyone's freedom to edit all of the common product. By comparing versions we can simultaneously evolve optimal contents and sensitively evaluate each individual's contributions to the team effort.

Accountable evaluation

Evaluation is a continuous editorial process, visible in the common product to all readers, and subject to the usual mechanisms of discussion and revision. Issues such as favoritism, bias, cultural astigmatism, etc. are dealt with organically, transparently, and respectfully, in a forum expressive of our common social values, rather than through inflexible and absurd procedural limitations that turn the whole richness of evaluative process into the sterile, archaic dichotomy of “papers” and “exams.”

Standards

Of course the narrative evaluation comprised of multiple editorial interactions—ranging from formal commentary on successive drafts of personally-authored essays, to revising edits on a collectively-authored page introducing a scanned and translated historical document, to slight comments in moderation of a thoughtfully-revised debate among multiple interlocutors—will have to be reduced eventually, under the watchful arrogance of the occupying forces, into a single grade. This process, which by its very nature is entirely without intellectual integrity and ethical propriety, is usually conducted in a standardless fashion, which renders it even more banal and even more infuriating. Counting correct answers to questions is only one partial measure of the more and less successful aspects of an individual's progress in learning. The most sought-after examination grade, for example, will almost always fail to distinguish between an unimaginative thinker with a superb memory for the material covered and a bold creative reasoner with a penchant for occasional inaccuracy. Sidney Carton, Sherlock Holmes, and Harry Blackmun alike will (if the grader is lucky enough to get it right on the basis of a single, usually anonymous, experience) meet incongruously at the top of the class.

This is a charming result, but from the scientific point of view it's detestable. If the occupation is to be dealt with at all, it should be on a basis that preserves self-respect. So there should be some standards for this fallacious measurement.

We measure here three qualities: effort, commitment, and improvement. Little need be said as to the first: That diligence is part of the craft we are teaching should be instilled as insistently as any other aspect of the business of representing clients. Whatever we may do with Carton, we do not tend to retain him. Commitment in any team, even a large one, is palpable, The nature of the student's intellectual engagement is best measured through the instructor's editing of the personal essays, where the student's choice of question and mode of pursuing both the first draft and the broad-scale revision (all of which can occur multiple times in a semester if one limits the article length to 750 or 1,000 words) defines the student's strengths and weaknesses sensitively enough to enable the instructor to make concrete suggestions for immediate improvement. Improvement is documented in all the versions of all the materials the student has collaborated in writing and editing, and is therefore subject to effective measurement.

Grades

Reduction of multi-variate qualitative measurements to grades implies an algorithm implementing a model for collapsing space by throwing away information. Each turns complex surfaces captured in the evaluations in the wiki into a single numerical score, by turning a series of qualitative judgments into quantities and then operating on a vector of quantities to produce the score. Algorithms used vary depending on course size and other factors, but for each course some transparent process generates a rank ordering. That rank ordering is then normalized by the Law School's “upperclass elective curve” for the relevant course size. (Even LawContempSoc is subject to the upperclass elective curve, for the information of those 1Ls who are already afraid of this curiously foolish but inoffensive device for curbing the non-problem of “grade inflation.”) Because no ranking algorithm fails to produce some misjudgments, I make adjustments based on qualitative judgments of effort, commitment and improvement, acting within the limits set by the Law School's curve deviation allowances, so that all grade distributions are as generous as, but no more generous than, the occupying forces allow.

Aspiration

No system of evaluation that culminates in a grade can escape complicity with dishonesty, or achieve pedagogical goals in a fully responsible manner. The replacement of grades by collaborative evaluations should be an early and uncontroversial aspect of our adjustment to new technologies of teaching. But the occupation drags on, as all the banal forms of injustice do. When students demand change they will get it. Until they do, the decadence of business as usual will remain undisturbed.

-- EbenMoglen - 26 Jun 2009