Collaborative Development of an Open Source Online Student Rating of Instruction System

 

* Ellen Yu Borkowski, Director, Academic Support, University of Maryland

* Tom Head, Chief of Staff & Director for Administration, Learning Technologies, Virginia Tech

* Aaron Zeckoski, Senior Research Engineer, CARET, University of Cambridge
 

Abstract

Virginia Tech, Maryland, Michigan, and Cambridge are collaborating on the development of an open source online student rating of instruction system that will be integrated with Sakai. This presentation will describe the collaborative development partnership, the features and advantages of online student rating systems, and the results of pilot studies.

 

 Evaluation Definitions

      Formative evaluation is conducted for the purpose of improving and developing teaching.  It is typically conducted in mid-semester to gather information for improving that particular course.

      Summative evaluation is conducted for the purpose of making a decision about promotion, tenure, or teaching awards.  It is conducted at the end of the semester.

 

Evaluation System

The Evaluation System is a tool that runs under Sakai version 2.2 or higher. The Evaluation System is used to conduct formative and summative (end of term) evaluations and review the results. An evaluation consists of a series of scaled and/or free text questions, which are delivered to students over a fixed time period (typically a few days).

 

The evaluation system allows colleges, departments, and instructors to manage their own body of questions which are part of a combined end of term evaluation. This allows colleges and departments within a larger institution to collect targeted information from their students, while preserving a consistent set of questions used through the institution.

 

Evaluation results can easily be reviewed within a single course, across colleges or departments, or across the entire university. Results can be reviewed for a single evaluation covering a short time period, or can be displayed for a series of evaluations conducted across a larger time period.

 

The evaluation system contacts students to notify them of upcoming evaluations automatically, and to remind students to complete an ongoing evaluation. The notification scheme is intended to increase the response rate.

 

Partners

University of Cambridge

University of Maryland

University of Michigan

Virginia Tech

 

Key Features

      Security - builds on Sakai permissions and security

      Validation - users may only complete evaluation once

      Reporting – view responses in html, or export to a spreadsheet

      Any instructor can give formative evaluations

      Integrated into Sakai classes and workspace to ensure visibility to students

      Templates can be used for multiple evaluations

      Diverse display options available at template creation

 

 Advantages (online versus paper)

      Instructors can easily add questions

      Greater flexibility - system supports ad hoc, mid-term, and end of term evaluations

      Improves efficiency - does not infringe on class time

      Rapid turnaround

      Higher quality student comments

      Cost savings - paper, processing time, handling and processing

      Response rate has been a concern at other universities

      Response rates in pilot studies at Virginia Tech have been comparable to paper-based system

   Any instructor can give formative evaluations

      University/Colleges/Departments can manage formative or summative evaluations

      University evaluations can be customized

      Email notification of evaluations for students and instructors

      One click access to evaluations for students

 

Current work being done

      University of Maryland, University of Michigan, Cambridge University and Virginia Tech collaboration

      Open source project designed to work within SAKAI framework

      Adding hierarchy to current tool to allow for multiple levels of question entry and report access

      Holistic approach used when redesigning system

 

Pilot results

     Virginia Tech

    10,000 students at Virginia Tech

    Response rates: 70 percent

                 University of Maryland

    Two pilots run over summer 2007

    Summer I: 5877 evaluations

    Summer II: 4481 evaluations

    Response rates: 40 percent

         University of Michigan

Approximately 546 classes Engineering mid-term evaluation pilot

 

University Adoptions

      University of Maryland

      University-wide Pilot Implementation in Fall 2007

      Pilot Hierarchy implementation in Spring 2008

      University of Michigan

      Pilot testing in Fall 2007

      Wider adoption in Winter 2008

      University of Cambridge

      Pilot in December 2007

      Plans to be in production in April 2008

 

More information:

http://confluence.sakaiproject.org/confluence/display/EVALSYS/