QA2 tag Results - Georgia Tech

Users

Conducted with Georgia Tech faculty over the course of three days, Feb. 3-5, 2009.

Demographics

Test Protocol

The new gradebook tool was deployed to a locally-branded Sakai instance. The gradebook was prepopulated with some sample data for 50 students and 20 or so assignments, tests and quizzes. Testers were asked to complete a variety of tasks and think out loud.

Test Script 

Notes We learned that this test script was too long and detailed. We never got through all the tasks. During testing we quickly adjusted to be looser and let the tester follow their nose, provided they kept talking about what they were doing and thinking. We steered the exploration back to particular tasks where the occasion seemed right, but this also meant that we could only get through 3 or 4 sets of tasks in the hour allotted. Since this initial test was meant to be an introductory survey of the interface to identify broader regions of difficulty, we expect that future rounds can be smaller and more focused to answer particular questions

User1 and User2 were tested with the QA1 tag, while User3 and User4 were tested with the QA2 tag.

Goals

This round of testing will focus on a general survey of pain points for gradebook 2 screens.

This test should discover:

• Can instructors accomplish their common tasks efficiently?
• Can instructors weight grades successfully?
• Can instructors navigate the various screens to find what they want?

Interaction Design

Results

All testers liked the initial appearance of the interface, but began to struggle, in varying degrees, when trying to weight and categorize grade items across the "Categories" and "Items" screens. As expected, the process of establishing formulas or business rules for how final grades are calculated proved most complex and confusing, a difficulty we'll likely not be able to reduce to zero, but it also seemed clear that the interface was not doing enough to make the process clear and discoverable. Navigation was also a problem. All were reluctant to guess where to go or didn't think the wording let them to the correct spot. They wanted back button rather than X's. They seemed to miss the visual clues for navigating and did not want to look outside the current screen area on any view.

Common Issues

Issues encountered by more than half of testers included:

• 3 out of 4 testers wanted to be able to quickly "tab" down the column entries to enter a series of grades for a single assignment using just the keyboard.
• 4 out of 4 testers did not understand what a "category" meant.
• 4 out of 4 testers did not understand what an "item" meant.
• 3 out of 4 testers could not discover how to return to the main table from the student view.
• 4 out of 4 testers tried to alter grade settings by double-clicking on column headings.
• Testers gravitated toward spending most of their time on the "Items" screen, because it contained the most information about both the "categories" and the "items" together in the same space.
• 4 out of 4 testers ignored the top-level menus ("New," "View," etc.) until specifically asked questions about them.
• 4 out of 4 testers tried to use the "history" pane to undo their mistakes.
• 4 out of 4 testers did not understand what the column drop-down menus were for.
• 4 out of 4 testers thought that "Deletes" should have more of a warning or way to quickly undo deletions.
• 4 out of 4 testers did not initially know what they were supposed to type in the "Drop Lowest" field, but after some hesitation 3 out of 4 finally tried to type in a number. The fourth tester tried to type in something else, but then said "It won't let me type in letters."
• 4 out of 4 testers looked for an "Add" button when trying to create new categories, items, or even students. With the QA2 tag a "New" menu was introduced, but even then it didn't register. The two testers who had a "New" menu still looked for an "Add" button closer to the main body of the tool where the tables/categories were.

On the positive side:

• 3 out of 4 testers found the context menus by right-clicking on grids.
• 4 out of 4 testers found out how to find individual student grade reports when asked to do so (and without being instructed how).

Particular comments by User:

User 1:

• "I don't know if this is for one student or all students" looking at the Item screen
• "I don't know what you mean by category."
• "Shouldn't there be some kind of warning when you delete an item?"
• "There should be a back button." This said at several points – student stats, history.
• ...

User 2:

• "These categories seem really cumbersome. I think I'd just use Excel for calculating formulas."
• "I think you'd want two gradebooks. One for the simplest cases, for most faculty. And then one for complex cases, and power-users."
• ...

User 3:

• "It's too easy to delete things, and too hard to create things."
• "I like that you can hide a whole category from the dropdown over columns."
• "What's a learner?"
• "I don't want my TA's to be able to delete anything let alone so easily" on the Item screen
• ...

User 4:

• "This really needs several warnings like "are you sure" before deleting"
• "I didn't want to click the X because I wasn't sure if it would close the whole browser"
• "It happens too fast. I want to be able to check several items and then press delete or include in grade."
• Actions? That's not clear. And we use import and export all the time, maybe they should be on the main screen?"
• ...