Noam Ebner, James Coben, and Christopher Honeyman, Editors. DRI Press, 2012. 306 pages.
By Todd Schenk
Many of us teaching negotiation pride ourselves on the innovative pedagogical tools we employ. Role-play simulation exercises, case studies, videos, and small-group vignettes are just some of the ways in which we actively and effectively engage students in the learning process. Yet we pay comparatively little attention to how we assess our students. Students expect useful feedback on their performance to effectively learn and grow, and for many of them, grades have consequences. In practice, our evaluations often fail to accurately reflect what students have learned, while conveying how they can improve. Of equal importance, our assessment schemas don’t always indicate what we want students to learn, yet implicitly and explicitly influence what they focus on. We owe it to them to do better.
Assessing Our Students, Assessing Ourselves steps into this void as a timely volume critiquing some of the common ways in which grading and evaluation are currently conducted, and introducing innovative approaches that negotiation teachers may employ to improve their practices. This is the third volume in the Rethinking Negotiation Teaching series published by the Dispute Resolution Institute at the Hamline University School of Law. It was edited by Noam Ebner, James Coben, and Christopher Honeyman, with contributions from more than 20 scholars. The genesis of the book was an international conference on negotiation pedagogy in Beijing in 2010.
The first two volumes in the Rethinking Negotiation Teaching series were:
- Rethinking Negotiation Teaching: Innovations for Context and Culture, and
- Venturing Beyond the Classroom.
Qualities Necessary for Effective Evaluation
In chapter two, Evaluating Our Evaluation: Rethinking Student Assessment in Negotiation Courses, Ebner, Yael Efron, and Kimberlee K. Kovach provide the big picture for the rest of the book. They conclude with a set of elements or qualities necessary for an effective evaluation system (page 34):
- Well-thought-out and theoretically grounded;
- Made very clear to students;
- Applied consistently by the teacher;
- Designed to elicit or promote learning, in addition to evaluating performance;
- Inclusive of different types of evaluation methods;
- Appreciative of technological and pedagogical advances in negotiation teaching and evaluation; and
- Carefully designed to tap and measure the course’s stated objectives in a valid and reliable manner.
While many of these prescriptions may seem obvious, the authors contend that they are not always followed. They acknowledge that a perfect assessment method will never be found, but that much more can be done to empirically improve on the current state of practice.
Pop Quizzes and Reflective Journals
The next four chapters examine tried-and-true assessment methods. Chapter three focuses on pop quizzes. While extremely common elsewhere in education, Ebner and Efron note their lack of use in negotiation courses, and suggest that teachers should take another look. Ebner and Efron assert that pop quizzes are useful when testing whether or not students know something presented as fact (page 43). Therein lies a key limitation of quizzes, which the authors acknowledge, but that should not be trivialized: many teachers perceive negotiation as much more of an art than a science, and thus do not see their material as a set of facts to be conveyed and regurgitated wholesale, but as propositions to be tested and discussed. Pop quizzes do not fit well into this paradigm. Nonetheless, Ebner and Efron argue that quizzes have merit as part of a broader evaluation system for various reasons, including their inherent objectivity and clarity of outcomes, ability to motivate students to learn content, direct educational value as students draw explicitly from what they have learned, and their ease of marking.
In contrast to quizzes, reflective journal assignments are commonly employed as evaluation methods in teaching negotiation. In chapter four, Bobbi McAdoo reviews their use, some of the challenges, and when and how reflective journal assignments might be most effective. She identifies challenges to this approach, including: the difficulties often felt in getting students to take journals seriously and engage in genuine reflection; the time it takes for teachers to adequately review and provide feedback; and the subjective nature of journal assignments and resulting difficulty in grading fairly. Nonetheless, journal assignments can facilitate reflective practice among students, motivating them to critically evaluate what they do and why.
Course participation is also regularly evaluated in negotiation courses. In chapter five, Ebner and Efron examine this method, taking particular issue with how ill defined and subjective it often is in practice. The range of what teachers even consider falling under the rubric of course participation varies, but might include attendance, the quality and quantity of comments made, disruptiveness, class presentations, and performance in negotiations. Some teachers acknowledge that they use course participation to even out inconsistencies, pushing up students they feel deserve a higher grade than they otherwise would have, and conversely to bump down others. The authors question whether evaluating course participation may be inherently problematic, but ultimately conclude with some questions they recommend that users of this tool ask themselves if they do intend to do so. As fraught with subjectivity as course participation may be, negotiation is an inherently participatory subject and thus, evaluating how actively students engage in class seems important to me. Relying solely on participation would certainly miss other objectives and unfairly penalize students who are shy or less able to engage; but perhaps some of the onus should be placed on typically quieter students to find ways to engage.
The Use of Video
Chapter six focuses on Using Video Recordings: A Mirror and a Window on Student Negotiation. Melissa Manwaring and Kovach promote this emerging technique as a powerful way to effectively capture and display key moments in students’ negotiations for their own self-reflection, and peer and teacher evaluation. The ubiquity of video capturing and editing technology makes this technique cheap and easy to use, and since students are used to being filmed, they are not typically disruptively nervous. In our own teaching at MIT, we have found videos to be a very powerful tool in the debriefing of exercises. The time it takes to edit and prepare short video clips should not, however, be underestimated, even when working with relatively user-friendly tools. We also encountered a challenge when a student refused to be taped – as recording becomes an increasingly prominent part of the assessment process, privacy concerns and how they can be respected will undoubtedly need to be addressed.
Student Assessment Criteria
The next five chapters focus on New Roles for Teachers, New Roles for Students. In chapter seven, Joel Lee makes a case for Negotiating the Assessment Criteria with students. In his own teaching, he has given students the task of negotiating among themselves the criteria that will be applied during the evaluation of forthcoming exercises. This approach responds to student concerns of fairness and subjectivity, and can have great educational value in and of itself, as students must reflect on what constitutes success or mastery in the subject matter they are learning. Even student attempts to game the system can lead to teachable moments. Students don’t always know what they should learn and thus negotiating their own assessment criteria isn’t always appropriate, but this method can engage students in deeper reflection and induce interesting negotiations.
In chapter eight, Boyd Fuller examines one-on-one Interviews as an Assessment Tool. He concludes that interviews offer an innovative way for teachers to work with students to assess how they are doing, identify their individual gaps, and provide learning opportunities in their own right. Fuller experimented with three different interview assessment methods in his own work: a mixed scenario approach, in which he engages in one or two mini-negotiations with each student, and has him or her analyze two or three small scenarios; simulated negotiation in which he spends the whole time in a two-party negotiation with each student; and coaching interview assessments in which he meets with students three times and has them reflect on their negotiation skill gaps and learning goals, devise experiments or strategies to fill these gaps, and ultimately reflect on what they learn via this process. Fuller concludes that the mixed scenario approach has substantial weaknesses, but that the simulated negotiation approach can be very useful in basic negotiation courses, and the coaching in advanced courses. One caveat that should be noted is the substantial instructor time this method necessitates, particularly in larger classes.
Student-Designed Role-Play Exercises
Role-play simulation exercises are widely used in negotiation pedagogy. In chapter nine, Ebner and Daniel Druckman propose a twist, suggesting that students learn more from designing exercises than from participating in them. While a believable assertion, the authors downplay the fact that effective exercise design can really only take place after a semester of using such games. As such, an exercise-design assignment might best be situated and understood as a capstone project in a course that has used exercises throughout. Furthermore, good exercise design requires serious attention to game mechanics beyond just the core teaching objectives. Students should not be misled into believing that exercises are easy to create and always work as planned. Having classes use at least some of the games designed by students may help to clear up any misconceptions.
Peer Evaluation of Outcomes
In chapter 10, Coben outlines an approach to peer evaluation he has experimented with that focuses not on participants’ performance during a simulation, but on the outcomes they reach. His students have typically endorsed this approach, and learned via the process of assessing each other. In chapter 11, Nancy A. Walsh explains how she has her students evaluate each other using a reputation index. Students are asked to nominate others who they feel have achieved the most positive and negative reputations as negotiators throughout the course of the semester, and provide explanations for their nominations. The total number of positive and negative nominations students receive impacts their final grades. She introduced this tool to counterbalance the inclusion of grades based strictly on the relative outcomes of negotiations. At the end of the day, reputation is certainly not everything, so Walsh recognizes the limitations of the index and thus weights it as a relatively small proportion of each student’s grade.
The final six chapters examine Special Tools for Special Contexts. In chapter 12, Sharon Press, Ebner, and Lynn P. Cohen explore how adventure learning may be assessed. “Adventure learning” is a catch-all term for experiential learning beyond the classroom, and might involve engaging in a real negotiation, participating in a role-play with real-world counterparts, or transferring negotiation skills to a real-world, non-negotiation context. Assessing such experiences can be challenging, raising ethical, cultural, and capacity issues. At this early stage, the authors suggest focusing on assessing adventure learning itself, rather than stifling creativity and creating problems by focusing on assessing student performance.
Melissa Nelken focuses on Evaluating Email Negotiations in chapter 13, reflecting on her experiences with email negotiations conducted among law school negotiation classes around the country. Negotiating by email is a relatively nascent tool in negotiation pedagogy that can demonstrate the challenges of multitasking and the limits of nuance, while providing students with an opportunity to negotiate with complete strangers rather than solely with classmates. Nelken recommends a mixed approach to evaluating email negotiations, combining direct evaluation of outcomes and indirect evaluation based on student reflection and debriefing.
Negotiation Competitions offer another vehicle for bringing students that do not know each other together to negotiate. In chapter 14, Nuno Delicado, Horacio Falcão, Ellen Deason, Press, Shahla Ali, Eric Blanchot, and Habib Chamoun-Nicolas examine the advantages and disadvantages of evaluating the outcomes of competitions versus the style and process that negotiators employ. They consider various factors, ranging from how to rate the outcomes, to how judges should be prepared, and what feedback they should be asked to provide. Examining a wide variety of competitions in various fields and countries, they conclude that there is no one-size-fits-all way of organizing and assessing competitions, but that organizers must be cognizant of the options and choose those that fit well with their goals.
In chapter 15, Falcão tackles the typically competitive win-lose nature of negotiation competitions. He proposes that these competitions be framed as challenges to more accurately reflect the collaborative tendencies and integrative approaches we want to teach students, rather than concentrating on who wins and who loses, which can focus too much attention on distributive techniques for capturing as much value as possible at others’ expense. Falcão proposes events that assess “self-improvement, not winning,” and pay as much attention to process as to outcomes.
Fuller and Sohni Kaur introduce A Benchmarking System for Assessment: An Experiment Creating More Transparency in Grading in chapter 16. Using this approach, students are tested against predetermined benchmarks of understanding of key concepts or skills. One approach uses accumulative benchmarks to provide students with an element of choice in what they focus on, allowing them to reach deeper parallel benchmarks in particular areas, rather than necessitating a focus on the breadth of benchmarks across subtopics. The authors’ experience with this method suggests that it takes work, but is very helpful in identifying more precisely what students have captured and where they need more teaching or assistance.
In chapter 17, Michelle LeBaron introduces portfolio evaluation. This approach is similar to the benchmarking system, but encourages “affective and sensory reflection” rather than merely cognitive understanding. Students compile assignments and other artifacts of learning into personal portfolios that document their learning and, of equal importance, induce reflection on the content, process, and interconnectedness of what they have learned.
In the epilogue, the editors acknowledge that the project of critiquing and devising best practices around how we can most effectively assess students in negotiation courses is still incomplete. This volume represents just a first step into an underexplored, yet critically important, part of our pedagogy. Some approaches, including the use of class presentations, are left unexplored in the volume. Other proposals, such as the shift from negotiation competitions to challenges are too nascent to allow for thorough evaluation based on repeated use. However, the editors and other authors deserve kudos for raising assessment as an issue. May the dialogue continue and efforts to improve practice expand.
Assessing Our Students, Assessing Ourselves is available via the PON Clearinghouse.