Vogel, D.R., Van Genuchten, M., Lou, D., Van Ekkout, M, Adams, A. (2001). Exploratory research on the role of national and professional cultures in a distributed learning project. IEEE Transactions on Professional Communication 44(2), 114-125.
While according to Neff and Whithaus (2008) process scripts can be used effectively in any type of distance learning activity, this article by Vogel et al. (2001) shows how it is especially important to provide a detailed process description to students working in intercultural virtual teams. The article describes a seven-week project where ten different heterogeneous teams of students from the Netherlands and from Hong Kong created a report with a topic in software engineering.
The researchers identify in the study that the two main factors associated with successful team work are team feeling and trust. Team feeling can be defined as the overall impression of team members that they truly feel like an integral part of the team. While Dutch students reported significantly lower rates of team feeling than their Hong Kong counterparts in this study, overall team feeling numbers were much higher in high-performing teams. The rates of reported trust levels were also lower with Dutch students, but a correlation was seen between high frequency of communication and increased trust levels. Vogel et al. attribute the differences in the reported rates to cultural differences; Dutch students took the initiatives, and were more concerned with their individual performance than students in Hong Kong with a collectivistic culture background who waited and left room for others to come up with ideas. The different cultural approach to the project could have been influenced towards a more positive outcome by instructors requiring a certain level of input (i.e. posts on discussion boards) by a certain date form all team members. This could have motivated Hong Kong team members to participate equally in the initial stage, which would have resulted in higher trust by their Dutch team mates.
When addressing the lessons learned from the project, the article by Vogel et.al emphasizes the importance of support instructors need to give throughout the project which includes quick response time, frequent monitoring of progress, and making sure that technology works for all participants while using a supportive rather than directive management style. In addition, a “sandwich structure” (p 122) is recommended to maximize effectiveness in the process of creating a collaborative report which means starting out the project with synchronous interaction, continuing asynchronously and finishing up with synchronous meetings.
This study contains many practical ideas that can be applied to cross cultural team work. While the article focuses on a software project, any type of writing class that requires students to collaborate on creating a document can use the suggestions that grew out of this extensive study. While the teams were made up of members from only two different cultures, the overall lessons learned could be applied to any diverse group. I especially agree with the authors in recommending the “sandwich structure” to control the group process as it has been suggested by other authors (i.e. Grosse 2002, Berry 2006) reviewed in previous blog entries that face-to-face meetings are crucial in the decision making phase of the project.
References
Berry, G. R. (2006). Can computer-mediated asynchronous communication improve team process and decision making? Learning from the management literature. Journal of Business Communication 43(4), 344-366.
Grosse, C. U. (2002). Managing communication within virtual intercultural teams. Business Communication Quarterly 65(4), 22-38.
Neff, J.M. & Whithaus, C. (2008) Writing Across Distances & Disciplines: Research and Pedagogy in Distributed Learning. New York: Lawrence Erlbaum Associates.
Wednesday, June 9, 2010
Sunday, June 6, 2010
Pedagogical Tool Review
Web-based writing assessment tools: Rubistar and Waypoint
Computers and Writing Assessment
While assessment methods have never been the center of attention in writing pedagogy, a considerable amount of effort has been made to align assessment methods with current writing pedagogy. Although the results of writing instruction cannot be measured with objective multiple choice tests, initiatives to develop holistic scoring that makes writing pieces comparable based on a normative scale keep appearing constantly, and can be seen in the assessment of the writing portion of standardized tests. However, as Peter Elbow (1996) points out, these are futile efforts that base the true score of an essay with the normalized reader. Applying computers as readers has also been tried with varying degrees of success (Hout 1996, Wohlpart et al. 2008).
As the trajectory of the development of writing pedagogy moves away from error correction and focuses on the collective constructive process instead of the product, most writing instructors still need to assign grades at the end of each semester. While the use of portfolios in grading is one way to adjust assessment with pedagogy, rubrics have also been widely used to establish grading criteria that bring the subjective process of grading essays as close as possible to the objective assessment expected by students and administration.
Rubistar
The web-based tool, Rubistar, provides an easy way to create rubrics, or use and customize one of the thousands of searchable rubrics already stored on the site. Rubistar, a venture of the Advanced Learning Technologies project at the University of Kansas Center for Research on Learning can be found at http://rubistar.4teachers.org/index.php and is free to use for anyone after they have registered with the site. While many rubrics on Rubistar are aimed at K-12 teachers, there are some that were created for freshman composition assignments and for professional writing courses. Furthermore, there are templates that make designing a rubric very easy. These templates are grouped thematically under such categories as oral projects, science, work skills, research and writing. Once a template is selected, a basic template becomes visible that has room for 10 rubric categories each evaluated on a four-point scale. The rubric categories can be selected from a list of evaluation criteria and can be modified if necessary. Also the text describing each of the four scales can be customized. The resulting rubric will automatically fit on a printed page and can be shared with students either in its printed form or via the Internet. Once the rubric has been saved in the database, it can be retrieved and printed any time from the web.
Rubistar also provides the option to analyze assessment results for each assignment that has been evaluated with the same rubric. In its analytical mode, instructors can enter for each scale level in a given rubric how many students have received that grade. While the website claims that this is a good way for instructors to determine which areas need improvement for many students, the analytical tool of Rubistar is rather simplistic, instructors can come up with the same statistics using paper and pencil and half the time needed to fill in the analytical rubrics on Rubistar.
Some of the thematically organized templates can be used effectively in teaching writing online. While instructors have to make sure that the learning outcomes they identified for the course and for each assignment are aligned with the criteria specified in the rubric, the fact that students can access these rubrics using the URL posted by the instructor makes it a valuable time saving tool in the assessment of papers. Especially when these rubrics are shared with students before an assignment is due or used for peer-review, instructors’ grading practices will be demystified and students will be able to revise their essays more effectively (Warnock 2009a, Wohlpart et al. 2008). Using rubrics and sharing them ahead of time helps students to set clear goals for themselves, gives them the ability to “self-assess”, and provides them with the vocabulary they need to critically analyze their writing (Arter and McTighe, 2001, p 12).
In addition to the rubrics that are on the website and those that can be created to evaluate writing pieces, the site also contains rubrics to evaluate oral presentations and collaborative work. These can often be used in Professional Writing classes where oral presentations and collaborative projects are the norm. For example, the rubric used to evaluate collaborative working skills can become an inherent part of the group project used by group members to evaluate each other’s performance.
What is the most beneficial feature of Rubistar? Long before such tools were widely available on the Internet, Hout (1996) has already praised the potential of computer based assessment that enables many users to share rubrics across local and global networks. Rubistar makes this sharing possible and especially since it has been integrated with another assessment tool, Waypoint, it can serve as a basis for providing feedback to students that not only helps students to understand the instructor’s grading criteria but also enables them to revise their work using detailed, constructive comments.
Waypoint Outcomes
While Waypoint also uses rubrics as the basis for student writing evaluation, it differs from Rubistar not only in the many additional features it contains, but also because it is not a free tool, but can be accessed only if your institution purchases the software. Once adapted for institutional use, the benefits are many, claims Subjective Metrics, Inc. the developer of the Waypoint Outcomes software (http://waypointoutcomes.com).
Waypoint outcomes has been developed to “help educators improve the quality of feedback to students on authentic tasks and build a culture of continuous improvement” (Waypoint Outcomes 2010). The basis of the web-based software is a rubric system that can be developed within the program or can be imported from other programs such as Rubistar. The rubrics than can be used to evaluate written assignments in many different fields, such as business, engineering, nursing or any other field where instructors assess students’ problem solving, writing, or critical thinking skills. Goodrich (2008) claims that the more detailed feedback students get through Waypoint helps develop their analytical and critical thinking skills.
When used for the evaluation of individual assignments, instructors check the appropriate rubric on a professionally formatted feedback sheet, and have the opportunity to add individual comments, paste an example segment from student text, or refer students to a specific page in the handbook for further instruction on how to correct a problem or improve a specific feature of their writing. This feedback sheet is easy to fill out, and has a color coded system for observation, recommendation, and reference. Because Waypoint can also be integrated with Blackboard, this feedback sheet can be sent to Blackboard grade book and can be viewed by students through Blackboard together with the grade they received for an assignment. This is a great feature that can be very beneficial in distance education. Students are able to see instant feedback when they check on their grade, and can use the evaluation report to improve their writing.
Waypoint can be used not only for individual assessment, but also to evaluate and compare courses within a major, an academic unit or even a whole university. It has been shown valuable for accreditation purposes for engineering schools, colleges of education and business schools. Warnock (2009b) for example, describes how Drexel University designed an accreditation focused evaluation program where papers written in a writing-intensive business course were evaluated by two different readers using a ten category rubric supported by Waypoint. The purpose of these evaluations was not to give feedback to students but it was aimed at establishing whether most students achieve the desired learning outcomes across the course which would prove the effectiveness of instruction for accreditation purposes.
Waypoint also has an analytical component, but it is much more sophisticated than that of Rubistar’s. Using Waypoint, faculty and administrators can easily aggregate data from different courses which will be displayed as graphs for easy comparison. This same data-analytical component can also be used to compare distance learning courses with on-site courses in their effectiveness of meeting learning outcomes. Within a class, individual instructors can easily identify areas which need to be revisited based on the analytical results. This is especially helpful for distance learning instructors who cannot read from nonverbal signs when students have difficulty with a specific concept.
Peer review is also an area where Waypoint can be used successfully in distance education. When teaching writing online, keeping track of the peer review process can be difficult, especially when teaching a large number of students. Using Waypoint, students can provide insightful feedback while instructors can monitor whether students have done their peer review on time and are able to access each review from a simple table that collects all peer-review data (McCann 2008). Here, instructors can also see the average score students receive through peer-review and thus can identify drafts with very low scores and can reach out to those students early on.
All of these features make Waypoint Outcomes an excellent tool to be used in teaching writing from a distance. While not all instructors might have access to Waypoint because it is only available through institutional subscription, sharing rubrics with students before an assignment is due and for peer review purposes in other ways, for example through Rubistar, enhances student performance in both on-site and distance writing courses.
References
Arter, J. & Mc. Tighe, J. (2001). Scoring Rubrics in the Classroom. Using Performance Criteria for Assessing and Improving Student Performance. Thousand Oaks, CA: Corwin Press.
Elbow, P. (1996). Writing assessment in the 21st century: A utopian view. In L. Bloom, D. Daiker, & E. White (Eds.), Composition in the twenty-first century: Crisis and change (pp. 83-100). Carbondale: Southern Illinois University Press.
Goodrich C (2008). Using web-based software to enhance student learning of analytical and critical skills. Journal of Educational Technology Systems 36(3), 247-253.
Hout, B. (1996). Computers and assessment: Understanding two technologies. Computers and Composition 13(2), 231-243.
McCann, A. (2008). Effective peer review: Leveraging the learning management system. Retrieved June 5, 2010 from http://docs.subjectivemetrics.com/home/aboutus/press/Effective%20Peer%20Review.pdf
Waypoint Outcomes (2010) Retrieved June 6, 2010 from http://waypointoutcomes.com
Warnock, S. (2009a). Teaching Writing Online: How and Why? Urbana, IL: National Council of Teachers of English
Warnock, S. (2009b). Methods and results of an accreditation-driven writing assessment in a business college. Journal of Business and Technical Communication 23, 83-107.
Wohlpart, A. J., Lindsey, C., Rademacher, C. (2008). The reliability of computer software to score essays: Innovations in a humanities course. Computers and Composition 25(2), 203-223.
Computers and Writing Assessment
While assessment methods have never been the center of attention in writing pedagogy, a considerable amount of effort has been made to align assessment methods with current writing pedagogy. Although the results of writing instruction cannot be measured with objective multiple choice tests, initiatives to develop holistic scoring that makes writing pieces comparable based on a normative scale keep appearing constantly, and can be seen in the assessment of the writing portion of standardized tests. However, as Peter Elbow (1996) points out, these are futile efforts that base the true score of an essay with the normalized reader. Applying computers as readers has also been tried with varying degrees of success (Hout 1996, Wohlpart et al. 2008).
As the trajectory of the development of writing pedagogy moves away from error correction and focuses on the collective constructive process instead of the product, most writing instructors still need to assign grades at the end of each semester. While the use of portfolios in grading is one way to adjust assessment with pedagogy, rubrics have also been widely used to establish grading criteria that bring the subjective process of grading essays as close as possible to the objective assessment expected by students and administration.
Rubistar
The web-based tool, Rubistar, provides an easy way to create rubrics, or use and customize one of the thousands of searchable rubrics already stored on the site. Rubistar, a venture of the Advanced Learning Technologies project at the University of Kansas Center for Research on Learning can be found at http://rubistar.4teachers.org/index.php and is free to use for anyone after they have registered with the site. While many rubrics on Rubistar are aimed at K-12 teachers, there are some that were created for freshman composition assignments and for professional writing courses. Furthermore, there are templates that make designing a rubric very easy. These templates are grouped thematically under such categories as oral projects, science, work skills, research and writing. Once a template is selected, a basic template becomes visible that has room for 10 rubric categories each evaluated on a four-point scale. The rubric categories can be selected from a list of evaluation criteria and can be modified if necessary. Also the text describing each of the four scales can be customized. The resulting rubric will automatically fit on a printed page and can be shared with students either in its printed form or via the Internet. Once the rubric has been saved in the database, it can be retrieved and printed any time from the web.
Rubistar also provides the option to analyze assessment results for each assignment that has been evaluated with the same rubric. In its analytical mode, instructors can enter for each scale level in a given rubric how many students have received that grade. While the website claims that this is a good way for instructors to determine which areas need improvement for many students, the analytical tool of Rubistar is rather simplistic, instructors can come up with the same statistics using paper and pencil and half the time needed to fill in the analytical rubrics on Rubistar.
Some of the thematically organized templates can be used effectively in teaching writing online. While instructors have to make sure that the learning outcomes they identified for the course and for each assignment are aligned with the criteria specified in the rubric, the fact that students can access these rubrics using the URL posted by the instructor makes it a valuable time saving tool in the assessment of papers. Especially when these rubrics are shared with students before an assignment is due or used for peer-review, instructors’ grading practices will be demystified and students will be able to revise their essays more effectively (Warnock 2009a, Wohlpart et al. 2008). Using rubrics and sharing them ahead of time helps students to set clear goals for themselves, gives them the ability to “self-assess”, and provides them with the vocabulary they need to critically analyze their writing (Arter and McTighe, 2001, p 12).
In addition to the rubrics that are on the website and those that can be created to evaluate writing pieces, the site also contains rubrics to evaluate oral presentations and collaborative work. These can often be used in Professional Writing classes where oral presentations and collaborative projects are the norm. For example, the rubric used to evaluate collaborative working skills can become an inherent part of the group project used by group members to evaluate each other’s performance.
What is the most beneficial feature of Rubistar? Long before such tools were widely available on the Internet, Hout (1996) has already praised the potential of computer based assessment that enables many users to share rubrics across local and global networks. Rubistar makes this sharing possible and especially since it has been integrated with another assessment tool, Waypoint, it can serve as a basis for providing feedback to students that not only helps students to understand the instructor’s grading criteria but also enables them to revise their work using detailed, constructive comments.
Waypoint Outcomes
While Waypoint also uses rubrics as the basis for student writing evaluation, it differs from Rubistar not only in the many additional features it contains, but also because it is not a free tool, but can be accessed only if your institution purchases the software. Once adapted for institutional use, the benefits are many, claims Subjective Metrics, Inc. the developer of the Waypoint Outcomes software (http://waypointoutcomes.com).
Waypoint outcomes has been developed to “help educators improve the quality of feedback to students on authentic tasks and build a culture of continuous improvement” (Waypoint Outcomes 2010). The basis of the web-based software is a rubric system that can be developed within the program or can be imported from other programs such as Rubistar. The rubrics than can be used to evaluate written assignments in many different fields, such as business, engineering, nursing or any other field where instructors assess students’ problem solving, writing, or critical thinking skills. Goodrich (2008) claims that the more detailed feedback students get through Waypoint helps develop their analytical and critical thinking skills.
When used for the evaluation of individual assignments, instructors check the appropriate rubric on a professionally formatted feedback sheet, and have the opportunity to add individual comments, paste an example segment from student text, or refer students to a specific page in the handbook for further instruction on how to correct a problem or improve a specific feature of their writing. This feedback sheet is easy to fill out, and has a color coded system for observation, recommendation, and reference. Because Waypoint can also be integrated with Blackboard, this feedback sheet can be sent to Blackboard grade book and can be viewed by students through Blackboard together with the grade they received for an assignment. This is a great feature that can be very beneficial in distance education. Students are able to see instant feedback when they check on their grade, and can use the evaluation report to improve their writing.
Waypoint can be used not only for individual assessment, but also to evaluate and compare courses within a major, an academic unit or even a whole university. It has been shown valuable for accreditation purposes for engineering schools, colleges of education and business schools. Warnock (2009b) for example, describes how Drexel University designed an accreditation focused evaluation program where papers written in a writing-intensive business course were evaluated by two different readers using a ten category rubric supported by Waypoint. The purpose of these evaluations was not to give feedback to students but it was aimed at establishing whether most students achieve the desired learning outcomes across the course which would prove the effectiveness of instruction for accreditation purposes.
Waypoint also has an analytical component, but it is much more sophisticated than that of Rubistar’s. Using Waypoint, faculty and administrators can easily aggregate data from different courses which will be displayed as graphs for easy comparison. This same data-analytical component can also be used to compare distance learning courses with on-site courses in their effectiveness of meeting learning outcomes. Within a class, individual instructors can easily identify areas which need to be revisited based on the analytical results. This is especially helpful for distance learning instructors who cannot read from nonverbal signs when students have difficulty with a specific concept.
Peer review is also an area where Waypoint can be used successfully in distance education. When teaching writing online, keeping track of the peer review process can be difficult, especially when teaching a large number of students. Using Waypoint, students can provide insightful feedback while instructors can monitor whether students have done their peer review on time and are able to access each review from a simple table that collects all peer-review data (McCann 2008). Here, instructors can also see the average score students receive through peer-review and thus can identify drafts with very low scores and can reach out to those students early on.
All of these features make Waypoint Outcomes an excellent tool to be used in teaching writing from a distance. While not all instructors might have access to Waypoint because it is only available through institutional subscription, sharing rubrics with students before an assignment is due and for peer review purposes in other ways, for example through Rubistar, enhances student performance in both on-site and distance writing courses.
References
Arter, J. & Mc. Tighe, J. (2001). Scoring Rubrics in the Classroom. Using Performance Criteria for Assessing and Improving Student Performance. Thousand Oaks, CA: Corwin Press.
Elbow, P. (1996). Writing assessment in the 21st century: A utopian view. In L. Bloom, D. Daiker, & E. White (Eds.), Composition in the twenty-first century: Crisis and change (pp. 83-100). Carbondale: Southern Illinois University Press.
Goodrich C (2008). Using web-based software to enhance student learning of analytical and critical skills. Journal of Educational Technology Systems 36(3), 247-253.
Hout, B. (1996). Computers and assessment: Understanding two technologies. Computers and Composition 13(2), 231-243.
McCann, A. (2008). Effective peer review: Leveraging the learning management system. Retrieved June 5, 2010 from http://docs.subjectivemetrics.com/home/aboutus/press/Effective%20Peer%20Review.pdf
Waypoint Outcomes (2010) Retrieved June 6, 2010 from http://waypointoutcomes.com
Warnock, S. (2009a). Teaching Writing Online: How and Why? Urbana, IL: National Council of Teachers of English
Warnock, S. (2009b). Methods and results of an accreditation-driven writing assessment in a business college. Journal of Business and Technical Communication 23, 83-107.
Wohlpart, A. J., Lindsey, C., Rademacher, C. (2008). The reliability of computer software to score essays: Innovations in a humanities course. Computers and Composition 25(2), 203-223.
Wednesday, June 2, 2010
Blog #4
Artiz, J. (2010). Cognitive organization and identity maintenance in multicultural teams. Journal of Business Communication 47(1) 20-41.
When teaching writing courses, we often assume that we need to focus on written communication only while we design an assignment or course. However, when planning for a collaborative writing project, speaking, specifically meeting-talk, might be important to pay attention to especially at the beginning stages of a group writing project when ideas are considered and decisions are made about what direction the group should proceed. This issue becomes even more complex, when such projects take place in an online course.
While the talk of meetings in general has been observed in culturally homogenous groups, this article by Jolanta Artiz concentrates on culturally heterogeneous groups and observes that group members alter their conversational styles and strategies depending on the cultural makeup of the team. In this study, Artiz analyzes the discourse patterns of small teams made up of US-born and East-Asian-born English speakers. Her findings indicate that the US-born English speakers dominate discussions and even increase their conversational presence in groups where they are minorities. On the other hand, East Asian English speakers take even fewer turns when they are in the minority as to signal their different cultural background. Artiz points out that educators need to be aware of these trends and need to inform their students of such tendencies, so that they become more conscious of their speech patterns that can have a direct influence on group participation and on the outcome of the project.
How does this apply to an online course? Instructors who are convinced that certain phases of the group project can be more productive when approached through synchronous communication, might arrange videoconferencing or synchronous chat sessions for each group. If videoconferencing is used, rules of spoken discourse definitely apply. However, if groups engage in synchronous chat sessions, additional factors need to be taken into account. Markman (2009) highlights the differences between conversations and chats. She explains that while we perceive online chats as very similar to face-to-face conversations, the turn-taking structure of the conversation is completely ambiguous as each speaker’s turn appears in the order that the server receives the entry. In addition, no one can see a turn until it is complete. These qualities of online chat make it more difficult and require more turns and attempts on the side of several participants to start and end meetings or to change topic. To avoid such problems, and to make meetings more productive in chat sessions, Markman suggests that groups create an agenda for each meeting, which will serve as a shared organizing principle for all members. In addition, appointing a chair who calls meetings might be a good idea, as it works very well in corporate settings. However, the effectiveness a leader in the educational group setting to keep the group focused has not been established in this study.
What all writing instructors can take away from this article is that we cannot neglect the guiding principles of synchronous interaction, be it face-to-face conversation or text chat, because it has a significant effect on how successful collaborative group projects will be in our online classes.
References
Markman, K. (2009). So what shall we talk about? Openings and closings in shat based virtual meetings. Journal of Business Communication 46(1), 150-170.
When teaching writing courses, we often assume that we need to focus on written communication only while we design an assignment or course. However, when planning for a collaborative writing project, speaking, specifically meeting-talk, might be important to pay attention to especially at the beginning stages of a group writing project when ideas are considered and decisions are made about what direction the group should proceed. This issue becomes even more complex, when such projects take place in an online course.
While the talk of meetings in general has been observed in culturally homogenous groups, this article by Jolanta Artiz concentrates on culturally heterogeneous groups and observes that group members alter their conversational styles and strategies depending on the cultural makeup of the team. In this study, Artiz analyzes the discourse patterns of small teams made up of US-born and East-Asian-born English speakers. Her findings indicate that the US-born English speakers dominate discussions and even increase their conversational presence in groups where they are minorities. On the other hand, East Asian English speakers take even fewer turns when they are in the minority as to signal their different cultural background. Artiz points out that educators need to be aware of these trends and need to inform their students of such tendencies, so that they become more conscious of their speech patterns that can have a direct influence on group participation and on the outcome of the project.
How does this apply to an online course? Instructors who are convinced that certain phases of the group project can be more productive when approached through synchronous communication, might arrange videoconferencing or synchronous chat sessions for each group. If videoconferencing is used, rules of spoken discourse definitely apply. However, if groups engage in synchronous chat sessions, additional factors need to be taken into account. Markman (2009) highlights the differences between conversations and chats. She explains that while we perceive online chats as very similar to face-to-face conversations, the turn-taking structure of the conversation is completely ambiguous as each speaker’s turn appears in the order that the server receives the entry. In addition, no one can see a turn until it is complete. These qualities of online chat make it more difficult and require more turns and attempts on the side of several participants to start and end meetings or to change topic. To avoid such problems, and to make meetings more productive in chat sessions, Markman suggests that groups create an agenda for each meeting, which will serve as a shared organizing principle for all members. In addition, appointing a chair who calls meetings might be a good idea, as it works very well in corporate settings. However, the effectiveness a leader in the educational group setting to keep the group focused has not been established in this study.
What all writing instructors can take away from this article is that we cannot neglect the guiding principles of synchronous interaction, be it face-to-face conversation or text chat, because it has a significant effect on how successful collaborative group projects will be in our online classes.
References
Markman, K. (2009). So what shall we talk about? Openings and closings in shat based virtual meetings. Journal of Business Communication 46(1), 150-170.
Subscribe to:
Posts (Atom)