Pedagogical Tool Review

Web-based writing assessment tools: Rubistar and Waypoint


Computers and Writing Assessment

While assessment methods have never been the center of attention in writing pedagogy, a considerable amount of effort has been made to align assessment methods with current writing pedagogy. Although the results of writing instruction cannot be measured with objective multiple choice tests, initiatives to develop holistic scoring that makes writing pieces comparable based on a normative scale keep appearing constantly, and can be seen in the assessment of the writing portion of standardized tests. However, as Peter Elbow (1996) points out, these are futile efforts that base the true score of an essay with the normalized reader. Applying computers as readers has also been tried with varying degrees of success (Hout 1996, Wohlpart et al. 2008).
As the trajectory of the development of writing pedagogy moves away from error correction and focuses on the collective constructive process instead of the product, most writing instructors still need to assign grades at the end of each semester. While the use of portfolios in grading is one way to adjust assessment with pedagogy, rubrics have also been widely used to establish grading criteria that bring the subjective process of grading essays as close as possible to the objective assessment expected by students and administration.


Rubistar

The web-based tool, Rubistar, provides an easy way to create rubrics, or use and customize one of the thousands of searchable rubrics already stored on the site. Rubistar, a venture of the Advanced Learning Technologies project at the University of Kansas Center for Research on Learning can be found at http://rubistar.4teachers.org/index.php and is free to use for anyone after they have registered with the site. While many rubrics on Rubistar are aimed at K-12 teachers, there are some that were created for freshman composition assignments and for professional writing courses. Furthermore, there are templates that make designing a rubric very easy. These templates are grouped thematically under such categories as oral projects, science, work skills, research and writing. Once a template is selected, a basic template becomes visible that has room for 10 rubric categories each evaluated on a four-point scale. The rubric categories can be selected from a list of evaluation criteria and can be modified if necessary. Also the text describing each of the four scales can be customized. The resulting rubric will automatically fit on a printed page and can be shared with students either in its printed form or via the Internet. Once the rubric has been saved in the database, it can be retrieved and printed any time from the web.
Rubistar also provides the option to analyze assessment results for each assignment that has been evaluated with the same rubric. In its analytical mode, instructors can enter for each scale level in a given rubric how many students have received that grade. While the website claims that this is a good way for instructors to determine which areas need improvement for many students, the analytical tool of Rubistar is rather simplistic, instructors can come up with the same statistics using paper and pencil and half the time needed to fill in the analytical rubrics on Rubistar.
Some of the thematically organized templates can be used effectively in teaching writing online. While instructors have to make sure that the learning outcomes they identified for the course and for each assignment are aligned with the criteria specified in the rubric, the fact that students can access these rubrics using the URL posted by the instructor makes it a valuable time saving tool in the assessment of papers. Especially when these rubrics are shared with students before an assignment is due or used for peer-review, instructors’ grading practices will be demystified and students will be able to revise their essays more effectively (Warnock 2009a, Wohlpart et al. 2008). Using rubrics and sharing them ahead of time helps students to set clear goals for themselves, gives them the ability to “self-assess”, and provides them with the vocabulary they need to critically analyze their writing (Arter and McTighe, 2001, p 12).
In addition to the rubrics that are on the website and those that can be created to evaluate writing pieces, the site also contains rubrics to evaluate oral presentations and collaborative work. These can often be used in Professional Writing classes where oral presentations and collaborative projects are the norm. For example, the rubric used to evaluate collaborative working skills can become an inherent part of the group project used by group members to evaluate each other’s performance.
What is the most beneficial feature of Rubistar? Long before such tools were widely available on the Internet, Hout (1996) has already praised the potential of computer based assessment that enables many users to share rubrics across local and global networks. Rubistar makes this sharing possible and especially since it has been integrated with another assessment tool, Waypoint, it can serve as a basis for providing feedback to students that not only helps students to understand the instructor’s grading criteria but also enables them to revise their work using detailed, constructive comments.


Waypoint Outcomes

While Waypoint also uses rubrics as the basis for student writing evaluation, it differs from Rubistar not only in the many additional features it contains, but also because it is not a free tool, but can be accessed only if your institution purchases the software. Once adapted for institutional use, the benefits are many, claims Subjective Metrics, Inc. the developer of the Waypoint Outcomes software (http://waypointoutcomes.com).
Waypoint outcomes has been developed to “help educators improve the quality of feedback to students on authentic tasks and build a culture of continuous improvement” (Waypoint Outcomes 2010). The basis of the web-based software is a rubric system that can be developed within the program or can be imported from other programs such as Rubistar. The rubrics than can be used to evaluate written assignments in many different fields, such as business, engineering, nursing or any other field where instructors assess students’ problem solving, writing, or critical thinking skills. Goodrich (2008) claims that the more detailed feedback students get through Waypoint helps develop their analytical and critical thinking skills.
When used for the evaluation of individual assignments, instructors check the appropriate rubric on a professionally formatted feedback sheet, and have the opportunity to add individual comments, paste an example segment from student text, or refer students to a specific page in the handbook for further instruction on how to correct a problem or improve a specific feature of their writing. This feedback sheet is easy to fill out, and has a color coded system for observation, recommendation, and reference. Because Waypoint can also be integrated with Blackboard, this feedback sheet can be sent to Blackboard grade book and can be viewed by students through Blackboard together with the grade they received for an assignment. This is a great feature that can be very beneficial in distance education. Students are able to see instant feedback when they check on their grade, and can use the evaluation report to improve their writing.
Waypoint can be used not only for individual assessment, but also to evaluate and compare courses within a major, an academic unit or even a whole university. It has been shown valuable for accreditation purposes for engineering schools, colleges of education and business schools. Warnock (2009b) for example, describes how Drexel University designed an accreditation focused evaluation program where papers written in a writing-intensive business course were evaluated by two different readers using a ten category rubric supported by Waypoint. The purpose of these evaluations was not to give feedback to students but it was aimed at establishing whether most students achieve the desired learning outcomes across the course which would prove the effectiveness of instruction for accreditation purposes.
Waypoint also has an analytical component, but it is much more sophisticated than that of Rubistar’s. Using Waypoint, faculty and administrators can easily aggregate data from different courses which will be displayed as graphs for easy comparison. This same data-analytical component can also be used to compare distance learning courses with on-site courses in their effectiveness of meeting learning outcomes. Within a class, individual instructors can easily identify areas which need to be revisited based on the analytical results. This is especially helpful for distance learning instructors who cannot read from nonverbal signs when students have difficulty with a specific concept.
Peer review is also an area where Waypoint can be used successfully in distance education. When teaching writing online, keeping track of the peer review process can be difficult, especially when teaching a large number of students. Using Waypoint, students can provide insightful feedback while instructors can monitor whether students have done their peer review on time and are able to access each review from a simple table that collects all peer-review data (McCann 2008). Here, instructors can also see the average score students receive through peer-review and thus can identify drafts with very low scores and can reach out to those students early on.
All of these features make Waypoint Outcomes an excellent tool to be used in teaching writing from a distance. While not all instructors might have access to Waypoint because it is only available through institutional subscription, sharing rubrics with students before an assignment is due and for peer review purposes in other ways, for example through Rubistar, enhances student performance in both on-site and distance writing courses.



References

Arter, J. & Mc. Tighe, J. (2001). Scoring Rubrics in the Classroom. Using Performance Criteria for Assessing and Improving Student Performance. Thousand Oaks, CA: Corwin Press.

Elbow, P. (1996). Writing assessment in the 21st century: A utopian view. In L. Bloom, D. Daiker, & E. White (Eds.), Composition in the twenty-first century: Crisis and change (pp. 83-100). Carbondale: Southern Illinois University Press.

Goodrich C (2008). Using web-based software to enhance student learning of analytical and critical skills. Journal of Educational Technology Systems 36(3), 247-253.

Hout, B. (1996). Computers and assessment: Understanding two technologies. Computers and Composition 13(2), 231-243.

McCann, A. (2008). Effective peer review: Leveraging the learning management system. Retrieved June 5, 2010 from http://docs.subjectivemetrics.com/home/aboutus/press/Effective%20Peer%20Review.pdf

Waypoint Outcomes (2010) Retrieved June 6, 2010 from http://waypointoutcomes.com

Warnock, S. (2009a). Teaching Writing Online: How and Why? Urbana, IL: National Council of Teachers of English

Warnock, S. (2009b). Methods and results of an accreditation-driven writing assessment in a business college. Journal of Business and Technical Communication 23, 83-107.

Wohlpart, A. J., Lindsey, C., Rademacher, C. (2008). The reliability of computer software to score essays: Innovations in a humanities course. Computers and Composition 25(2), 203-223.