Karma (or inspiration) points for discussion assessment

Assessing discussion contributions, whether in an online or on-campus course, is a drag. I like to participate in the discussions too, and if I am responsible for judging the quality of each student's contribution I am distracted. Plus, I've always thought that it was inappropriate for me to be the sole judge of value. Luckily, karma points emerged as a teaching and assessment strategy that fit well with my teaching approach. (Note: I now refer to these as "inspiration points" because I had a student who was very offended by the use of the name "karma"...I did not want the name -- which was not critical to the process -- to get in the way of the purpose, described below.)

Karma points, an approach used by members of the online Slashdot.org community (and similar to the valuing process used by community members of Amazon and Ebay), involve students in the evaluation of the quality of discussion contributions. The idea behind karma points is that the learning community, not a moderator or an instructor, should be responsible for (1) determining the value of community members’ posting in terms of helping the community achieve specific goals, and (2) awarding those valued contributions.

To make karma points work, I give each student a certain number (e.g., three) of karma points that she or he can assign to valued discussion contributions within a certain timeframe (e.g., by week’s end if online, or by the end of the evening session if on-campus). Because the students are evaluating each other, I work with them at the beginning of the semester to establish criteria for determining “value” and then apply the criteria to their assessment of peers’ contributions and the creation of their own contributions. For example, karma point criteria may include sharing original ideas, writing clearly, presenting a coherent argument, providing evidence to support an argument, “listening” to others and incorporating their ideas and perspectives, and so on (see below for an example of criteria).

***********************************************************************************
Criteria for Karma Points

Here is how we will assign our allotment of karma points for each discussion:

0 points: Though you may have introduced an interesting idea or contributed to the discourse, it is not original enough, or is somehow unclear.
1 point: You provide a succinct, interesting, original, and well-documented argument or idea, or provide a useful link or pertinent fact.
2 points: Your contribution is creative and original, and compellingly argues a very clear point. You support your contribution with evidence.
3 points: An exceptional contribution to the discourse, one that really opens eyes and encourages a lively discussion/debate. Exemplary in all respects.

Ways to Improve Chances of Receiving Karma Points

• Choose provocative subject lines to make our postings stand out.
• Present our own perspectives.
• Write clearly.
• Construct an argument. Provide evidence, present a rationale that supports our positions, and reference the opinions of others, linking to supplementary evidence when appropriate.
• Open up debate by remembering that the best response is one that gets people thinking, and that makes them want to reply.
• Learn from others who have posted before us by reading through the posts and referring to appropriate posts in our own.

Rules for Assigning Karma Points

Only award karma points to those who have contributed significantly to the discussion – vote trading is unacceptable. Award karma points based on the quality of the message, irrespective of the content of the message – vote for exceptional messages even if you do not necessarily agree with the ideas presented.

************************************************************************************

In my experience, the community-centered focus of karma points improves the quality of each post during a discussion because students are more reflective and thoughtful about their responses, make sure their responses are supported by evidence, and work hard to provide value to the learning community by moving the discussion forward. The example below shows the quality of postings karma points inspires and how karma points are assigned to postings even if students share opposing views (e.g., Michael disagrees with Michelle’s perspective but still acknowledges the posting’s value to the overall discussion). By using karma points, I participate more in the discussion because students have taken over part or all of the evaluation role. The karma points students accumulate for their valued contributions to the discussion can be used to determine a score for class participation.




Online, karma points can be distrubuted in the subject line of a student's post -- e.g., Karma point to Rick. In an on-campus classroom, I pass out index cards and have students track their point distribution on the cards.

Side note: A couple of years ago I was teaching an online course in which I used karma points. A few weeks before the end of the course, my daughter decided to be born five weeks early. I wanted to complete the course, but needed to scale back on some of the activities, specifically my monitoring of the last couple of weeks of online discussion. I informed the students that they were welcome to continue the discussions but that they need not assign karma points because I would no longer be monitoring. I thought for sure they would discontinue all discussion and focus on completing their final projects. To my amazement, they continued the discussion until the end of the course, and continued to acknowledge each other’s contributions by awarding karma points! They clearly believed that there was value in the process and, in fact, told me so in the end-of-course evaluations.


Related posts in this blog:
Discussion ground rules
Don’t jump into discussions
Engaging quieter online students
Small groups reporting out to the large group?
Beyond debates and conversational roles (Protocols Part 1)
Structures for asynchronous online discussions (Protocols Part 2)
Structures for synchronous online discussions (Protocols Part 3)
Structures for small groups reporting out to whole group (Protocols Part 4)

Discussion ground rules

Although it may seem like an overly formal step, I have found it very helpful to have students set "rules of engagement" to guide how they will participate and contribute to discussions in both on-campus and online courses. I have students answer questions like:

• What is our definition of a respectful, balanced discussion?
  
• How will we determine in what order people speak?
  
• How do we feel about interrupting?
  
• What should we do if someone dominates the discussion?
  
• What should we do if we don't hear from everyone in the room?
  
• What should we do if we discuss something controversial or uncomfortable?
  
• What should we do if someone says something we don't like?
  

Working together the students develop their rules of engagement to guide our discussions. This is a formal document that everyone in the class agrees to. We review it often, and make adjustments when needed.

This activity helps students reflect on their own discussion habits, makes it clear to everyone what is expected during discussions, and empowers students to stick up for themselves and others when one of the rules is violated (e.g., someone dominates the discussion or keeps interrupting others). It also gives students a chance to practice being in a discussion before they have to participate in a discussion covering a course topic, allowing them to become more comfortable with their peers and the discussion format.

Related posts in this blog:
Some guidelines for discussion participation
Don’t jump into discussions
Engaging quieter online students
Small groups reporting out to the large group?
Karma (or inspiration) points for discussion assessment
Beyond debates and conversational roles (Protocols Part 1)
Structures for asynchronous online discussions (Protocols Part 2)
Structures for synchronous online discussions (Protocols Part 3)
Structures for small groups reporting out to whole group (Protocols Part 4)

Preparing for guest speakers

I think it is nice to have guest speakers in my classes because they offer different perspectives on topics, and are often much better equipped than I am to talk about how things happen and work in the professional world for which the students are preparing.

However, I have found that it can also lead to a lost opportunity if students don't prepare ahead of time. Guests are only with us for a limited amount of time, and they are coming to class on their own time as a service to the community of practice. It is no fun for guests to feel that their time and expertise is not appreciated. So, I have students prepare in one of two ways:

1. Prepare the top 5 questions. I usually have students work in teams of 3-4 to come up with a list of 3 questions they would like the guest to address. I collect the questions, compile them into one master list, and give them back to the students. Then I have the students create categories of questions. I have them reflect on whether the questions are best addressed by this particular guest, or better addressed by someone else or via another resource. This reflective activity usually leads to us eliminating several questions. The students then vote for their top 3 questions. I track the vote on the board. Often this vote narrows it down to the top 5 questions. If not, then I ask for volunteers to defend why we should select a particular question, or why we should drop it. Once we have the 5 questions, we forward them to the guest.
   
   What questions should students ask guest speakers? What I tell students is to think of questions that will lead to unique responses from the speakers -- responses they wouldn't find in a text or via a Google search. They can ask them about their particular jobs, research, areas of interest. For example, students can ask speakers to talk/tell a story about:
   
   • The challenges of successfully developing and implementing [a computer system, a new curriculum, a new incentive system for the sales force -- fill in the blank]
     
   • The most rewarding project they worked on, and why it was rewarding
     
   • The project they consider a classic "war story," why it was so difficult, and what they would do differently now
     
   • Why they became a [systems analyst, professor, HR director -- fill in the blank]
     
   • The future of their profession, and what they see as being cutting edge in 5, 10, 15 years
     
   
   The most enjoyable guest speakers tend to tell great stories, so I ask students to think of prompts and questions they can present to guest speakers that encourage dramatic sharing of past, current, and future events.
   
   
2. Determine the topic. Sometimes it makes more sense to interview a guest instead of asking her or him to prepare a presentation. When this is the case, I ask students to determine a focused topic that is shared with the guest in advance. Then, we do a similar activity as #1 where students develop their interview questions. They also select 1-3 students who will serve as the interviewers.
   

The added bonus of having the students prepare, is that we can share our questions/main topic with the guest speaker in advance, helping the guest prepare for her or his time with us.

Jazzing up the syllabus

Recently I read a great article called The Promising Syllabus from James Lang, who writes a column for the Chronicle of Higher Education. This article got me thinking about my syllabi: what I include, the order of content, the number of "professor says 'no'" statements, and so on. Bottom line, it made me realize why I hate going over the syllabus the first night of class...boring, irrelevant, pessimistic. How can I write a syllabus that gets students -- and me -- excited about the course?

One strategy I have used is to start with a recent position announcement for a job that students would be interested in. Then, I use the position announcement to describe what we will do in the course, and how the course activities will help them prepare to apply for positions like the ones in the announcement. Here is an example:


Another example is to start the syllabus with vignettes or questions that grab students' attention, and then go on to describe how the course will help students address the challenges presented in the vignettes or answer the questions posed. Here is an example:



And yet another example is to sprinkle fun activities throughout the syllabus to grab students' attention, for example:

Storytelling for presence

One of the concerns I had early on in my online teaching -- I teach online and on-campus courses -- was how to establish my own presence in the course so that students felt a connection with me. Along with a number of other strategies, the strategy that has had the biggest positive impact is storytelling. My most elaborate example is a digital story created using Premiere (but could be easily created in iMovie, or even PowerPoint). In this digital story, I share personal information about a challenging time (going through tenure) and about why I am a professor. It also provides insight on what I value as an educator, what I consider to be "good" teaching.

Here is that digital story:
http://www.augustcouncil.com/~jdunlap/JoniDunlapSOR.mov
(alternatively, http://www.augustcouncil.com/~jdunlap/movie)

Related posts in this blog:
Creating digital stories with VoiceThread
Flickr photo sets

Creating structures for effective groupwork

One instructional strategy that I use a lot in my online and on-campus courses is group projects, involving learners in collaboration, teamwork, peer review, and so forth. I like this strategy because:

• Group work can help counter the isolation some students may feel in an online course,
  
• Exposing students to multiple perspectives can open their eyes to diverse ideas,
  
• Students can achieve higher expectations with collegial support, and
  
• The quality of individual student work can be enhanced through collaboration.
  

However, when I don't provide some structure around group projects, specifically addressing how group work will be assessed, these projects can lead to a lot of problems.

One thing I have done to help teams function more effectively is to have them establish a formal agreement describing how the team will function. A “Rules of Engagement” contact is especially important for students who have had negative past experiences in which they had to cover for team members who did not contribute. In the contract, each team can determine what to do if a team member cannot fulfill his or her obligations. This contract should include information on:

• Who will post: Which team member will be responsible for posting the group’s deliverables?
  
• Leader or not: Some groups chose a leader to keep things moving, some groups change that leader each week (or every other week or whatever), and some groups decide not to do this.
  
• Communication and deadlines: How will they communicate with each other? How often? Will they set interim deadlines? Some groups like to set certain deadlines, such as initial work done by Wednesday, rewrites by Friday, and final posted Sunday.
  
• Equal contributions: How will work be distributed?
  
• Preferred work style: Some people like to get things done during the week and take the weekend off and others prefer the opposite. What is your style? How will you handle style differences?
  
• Not getting work done: What will the group do if one member cannot fulfill his or her obligations? Will that member be docked points, or can the member make up the work?
  
• Known problems: Issues you know will come up, how to handle dates you know you will be out of town?
  

See below for a simple team agreement.

*******************************************************************************
Team E Agreement

1. Will you have a leader who keeps the team on track during team assignments?

After a preliminary review of the projects, Team E has decided that it is best to have a different team leader for each of the weeks that have a team assignment. The team leader will be assigned a week in advance based on personal schedules at that time.

2. How do you prefer to work?

After reviewing individual work schedules, we have decided to assign a team leader a week in advance so that he/she can organize and assign tasks to be completed by the weekend before the due date. This will provide an adequate amount of time for completion and review by our teammates. The team leader will assign a preliminary due date prior to the actual due date for all to adhere. This offers flexibility to each team member to work on his/her task, yet hold him/her accountable to successful completion of his or her task by the due date.

3. Do you agree to provide timely, substantive feedback?

We all agree that it is important to provide feedback that is positive. This benefits the team and provides open communication. It is important to critique constructively…the team wants to create a superior product.

4. How will you handle a team member who does not do what he/she has agreed to do?

The team is comprised of professionals and we expect if an individual has a problem (technical or personal) hindering him or her from performing the assigned task, the team will be notified ASAP so the team can assist in the completion of that task. The team may advise, inform, break the task into smaller and more manageable parts, or assign to another team member. The team leader will be the focal point for managing these activities. At that point the team will also negotiate how the team member’s assignment score should be affected.

********************************************************************************

I then have the students use the team agreements to assess each group members’ contribution (including their own contribution) to the submitted paper or product. Below is an example of a tool students adopted, based on the team contract, to assess each other’s contributions to group projects. These assessments can have ramifications, such as if a group member receives less than 50 points on the Group Review Form from more than one group member then that student's points for the deliverable will be reduced by 20%. I have found that this empowers students to have a say in the point distribution on group projects. This review process also functions as an incentive for all group members to fulfill their obligations. And, this approach has some additional benefits:

• Because students complete Group Review Forms describing each group members’ contribution, it summarizes the project work and minimizes the amount of time an instructor needs to spend examining the posts in each group’s private workspace forum for clues to level and quality of each student's contribution to the project.
  
• It alerts me to specific group and group member issues, and provides data to use when addressing those issues. This allows me to address the issues quickly and efficiently.
  
• Often, when given the means and opportunity, students are very thoughtful and detailed about the feedback they provide to group members, and are very honest about their own contributions. Therefore, the students' reviews of group members and themselves provides me with useful comments that can be included in feedback to individual students.