OSP 2.7 Test Suites

System & Regression Test Cases

QA Testers should use the OSP Test Scripts as the source for formal QA testing of the OSP 2.7 tools, and fill out the following grid with status. It's equally important to document testing done where no issues have been found, as it is to document testing that uncovered problems.

(Add more rows as necessary)

Scripts for testing IU Enhancements

Test Scripts

Note: These scripts assume significant prior knowledge of the core functions of each tool.  Detailed step by step instructions are not provided. 

Instructions:  Replace "Tester Name" in the column header with your name.  When you have successfully completed the actions in the test script, place a  in the appropriate column.  A collection of sample files is available here.

Instructions:

Replace "Tester Name" in the column header with your name.  When you have successfully completed the action under multiple conditions, put an  in the appropriate cell below.  If you are unable to complete the action due to a  bug, create a Jira and then add a  to the cell as well the Jira ID.

Authoring Matrices:  Coordinators

Note: Be sure to check the effect of the actions below on users in other roles.

Actions	Tester Name	Sakai QA Release	Comments
Add a matrix	Sean on QA3
Import a matrix	Sean on QA3
Manage Site Associations (makes it possible to link assignments, matrix cells, and wizard pages in associated sites to matrix cells in the current site)	Lynn on QA3
Add sites (in order to associate a site with the Matrices tool, you must be a member of the site you want to associate)	Sean on QA3
Delete sites	Sean on QA3
Verify that matrix cells, wizard pages, and assignments in associated sites can be linked to published matrices in the current site	Lynn on QA3 matrix cells assignments wizards		SAK-17839: Cannot open published wizard. Able to create link as coordinator, but could not open wizard to view link in various roles.
Verify appropriate permissions for each role and that permissions are working properly	Lynn on QA 3		SAK-17850: General Feedback link unavailable until item specific feedback is added (not really a perms issue)
Export a matrix	Sean on QA3
Import the exported matrix	Sean on QA3
Import matrices created in generic Sakai 2.5 and 2.6.	Sean on QA3
Delete a matrix	Sean on QA3
Edit Matrix Properties
Add/edit  description	Sean on QA3
Add and select a new CSS	Sean on QA3
Change default status colors	Sean on QA3
Add rows and columns	Sean on QA3
Move rows and columns	Sean on QA3
Edit row and column labels & colors	Sean on QA3
Delete rows and columns from an unpublished matrix (should not be able to delete from published USED matrix)	Sean on QA3
Set progression	Sean on QA3
Set default forms	Sean on QA3
Select both published and unpublished forms	Sean on QA3
Verify that all coordinators can see selected unpublished forms	Sean on QA3
Set default evaluators	Sean on QA3
Enable/disable Hide evaluations from participants	Sean on QA3
Set default reviewers	Sean on QA3
Enable/disable participant feedback invitations (and verify as participant)	Sean on QA3
Test and verify all "Feedback Options" for general and item specific feedback.	Sean on QA3
Edit Cell
Add description	Sean on QA3
Edit initial status	Sean on QA3
Add custom style (CSS)	Sean on QA3
Enable/disable participant attachments	Sean on QA3
Add/edit/delete each type of guidance.  Verify that guidance labels do not display when field is empty.	Sean on QA3
If set, override participant forms defaults and select different forms.	Sean on QA3
Override default feedback forms and reviewers	Sean on QA3
Re-enable default feedback forms and reviewers	Sean on QA3
Enable/disable "Allow participant to send feedback invitations to additional reviewers"	Sean on QA3
Override default evaluation forms and evaluators	Sean on QA3
Re-enable default evaluation forms and evaluators	Sean on QA3
Preview a matrix and add test data	Sean on QA Lynn on QA3		SAK-17595 filed, verified, and closed.
Publish a matrix and verify that test data is deleted	Sean on QA3
Edit Cell (these steps only available after current site has been associated with itself or with other portfolio sites)
Create link to cells in same or other matrices	Sean on QA3
Edit links	Sean on QA3
Delete links	Sean on QA3
Edit or delete links after work has been submitted.	Sean on QA3
View associated items.	Sean on QA3
Change the status of a cell with the Manage Status link and verify appropriate color and access.	Sean on QA3
Change the status of a cell for a specific group and verify that it has the desired affect on the cell for all group members	Sean on QA3
Change the status of a cell for the entire site and verify that it has the desired affect on the cell for all site members	Sean on QA3
Turn each matrix permission on and off and verify that it has the described affect on your abilities.	Lynn on QA3

Using Matrices: Participants

Actions	Tester Name	Sakai QA Release	Comments
Working with Forms
Add	Lynn on QA3	2.7.0-b01
Select existing	Lynn on QA3	2.7.0-b01
Edit	Lynn on QA3	2.7.0-b01
Remove	Lynn on QA3	2.7.0-b01
View completed form by clicking on form name.	Lynn on QA3	2.7.0-b01
Working with Attachments
Add	Lynn on QA3	2.7.0-b01
From local storage device	Lynn on QA3	2.7.0-b01
Attach a copy from resources	Lynn on QA3	2.7.0-b01
Web URL	Lynn on QA3	2.7.0-b01
Remove	Lynn on QA3	2.7.0-b01
Verify that attachment widget is suppressed when "disable participant attachments" is checked in Cell Settings.	Lynn on QA3	2.7.0-b01
Working with Reflections
Add	Lynn on QA3	2.7.0-b01
Edit	Lynn on QA3	2.7.0-b01
Remove	Lynn on QA3	2.7.0-b01
View by clicking on completed form title	Lynn on QA3	2.7.0-b01
Request Feedback (verify that button does not appear if no feedback form is selected for cell )	Lynn on QA3	2.7.0-b01
If "Allow participant to send feedback invitations to additional reviewers" is on, verify that participant can select reviewers.	Lynn on QA3	2.7.0-b01
If "Allow participant to send feedback invitations to additional reviewers" is off, verify that participants cannot select reviewers	Lynn on QA3	2.7.0-b01
Submit for Evaluation	Lynn on QA3	2.7.0-b01
Verify that cells progress properly from ready to pending status	Lynn on QA3	2.7.0-b01
Verify that button does not appear if no evaluation form is selected for cell)=	Lynn on QA3	2.7.0-b01
Verify that cells have the appropriate status	Lynn on QA3	2.7.0-b01
Verify that you cannot edit pending and locked cells	Lynn on QA3	2.7.0-b01
Verify that the appropriate icons appear in cells with content	Lynn on QA3	2.7.0-b01
Verify that you receive a notification when new feedback is added	Lynn on QA3	2.7.0-b01
Verify that you can read feedback added by a reviewer	Lynn on QA3	2.7.0-b01
Verify that you receive a notification when a new evaluation is added	Lynn on QA3	2.7.0-b01
Verify that you can read evaluations added by an evaluator (unless "hide evaluations from participants is turned on".	Lynn on QA3	2.7.0-b01
Verify that work submitted via a "goal aware" tool linked to a cell shows up in the cell.	Lynn on QA3	2.7.0-b01
Turn each matrix permission on and off and verify that it has the described affect on your abilities.	Lynn on QA3	2.7.0-b01

Working with Matrices: Reviewers

Action	Tester Name	Sakai QA Release	Comments
Open a published matrix	Lynn on QA3	2.7.0-b01
Inspect the dropdown user list; you should only see members of your own group unless you uou have the "view all groups" permission	Lynn on QA3	2.7.0-b01
Try to open a cell in which you are not a reviewer or evaluator; you should not be able to open.	Lynn on QA3	2.7.0-b01
Open cell in which you've been defined as a reviewer.	Lynn on QA3	2.7.0-b01
Verify that you can see and open all participant content	Lynn on QA3	2.7.0-b01
Add general feedback.	Lynn on QA3	2.7.0-b01	SAK-17850: General Feedback link unavailable until item specific feedback is added (not really a perms issue)
Add feedback for specific items.	Lynn on QA3	2.7.0-b01
Edit a previously saved feedback form	Lynn on QA3	2.7.0-b01
Delete a previously saved feedback form	Lynn on QA3	2.7.0-b01
Verify that you can read feedback you created and that the data is properly formatted.	Lynn on QA3	2.7.0-b01
Try to read feedback created by other reviewers (try with and without the "Can view feedback created by another user" permission)	Lynn on QA3	2.7.0-b01
Verify that you receive email notification when a participant submits cell for review.	Lynn on QA3	2.7.0-b01
Verify that you receive an email when participant invites feedback.	Lynn on QA3	2.7.0-b01
Verify that cells contain no feedback-related labels or widgets when cell doesn't contain a feedback form (i.e., the coordinator never selected a form for feedback)	Lynn on QA3	2.7.0-b01
Verify that you can see and open content submitted via a "goal-aware" tool	Lynn on QA3	2.7.0-b01
Turn each matrix permission on and off and verify that it has the described affect on your abilities.	Lynn on QA3	2.7.0-b01

Working with Matrices: Evaluators

Action	Tester Name	Sakai QA Release	Comments
Open a published matrix	Lynn on QA3	2.7.0-b01
Inspect the dropdown user list; you should only see members of your own group (unless you have the "view all groups" perm.)	Lynn on QA3	2.7.0-b01
Try to open a cell in which you are not a reviewer or evaluator; you should not be able to open.	Lynn on QA3	2.7.0-b01
Open cell in which you've been defined as an evaluator:	Lynn on QA3	2.7.0-b01
Verify that you can see and open all participant content	Lynn on QA3	2.7.0-b01
Add and evaluation and then ...	Lynn on QA3	2.7.0-b01
And leave the cell status as pending	Lynn on QA3	2.7.0-b01
And set the cell status to complete	Lynn on QA3	2.7.0-b01
And set the cell status to returned	Lynn on QA3	2.7.0-b01
Verify that you receive an email notification when a cell is submitted for which you are an evaluator. You should not receive the email if the submitter is not in the same group as you.	Lynn on QA3	2.7.0-b01
Verify that you can read evaluations you created and that the data is properly formatted.	Lynn on QA3	2.7.0-b01
Verify that you can edit your own previously saved evaluations.	Lynn on QA3	2.7.0-b01
Verify that you can remove your own previously saved evaluations.	Lynn on QA3	2.7.0-b01
Try to read evalations created by other evaluators (try with and without the "Can view evaluations created by another user" permission)	Lynn on QA3	2.7.0-b01
Verify that cells contain no evaluation-related labels or widgets when cell doesn't contain a evaluation form (i.e., the coordinator never selected a form for feedback)	Lynn on QA3	2.7.0-b01
Verify that you can see and open content submitted via a "goal-aware" tool.	Lynn on QA3	2.7.0-b01
Turn each matrix permission on and off and verify that it has the described affect on your abilities.	Lynn on QA3	2.7.0-b01

Instructions:

Replace "Tester Name" in the column header with your name.  When you have successfully completed the action under multiple conditions, put an  in the appropriate cell below.  If you are unable to complete the action due to a  bug, create a Jira and then add a  to the cell as well the Jira ID.

Using Evaluations: Coordinators

Action	Tester Name	Sakai QA Version	Comments
Open the evaluations tool.	Lynn on QA3-US	2.7.0-b01
Verify that the Use permission enables the designated evaluators to view and evaluate pending matrix cells and wizard pages.	Lynn on QA3-US	2.7.0-b01
Verify that the name of matrix owners do not display if evaluator does not have the "Can view / access user list and cell owner" permission for the specified matrix.	Lynn on QA3-US	2.7.0-b01

Using Evaluations: Evaluators

Action	Tester Name	Sakai QA Version	Comments
Open the evaluations tool.	Lynn on QA3-US
Verify that you can see and open matrix cells and wizard pages for which you are a designated evaluator
Verify that you cannot see matrix cells and wizard pages for which you are not a designated evaluator
Verify that you can only see owners who are in the same group you are if you do not have the "view all groups permission" in matrices.
Verify that you can see and filter on groups if you have the "view all groups" permission in matrices or are a member of 2 or more groups.
Open and evaluate pending items.
Verify that you can see and open all content submitted directly and indirectly (via linked goal aware tools) by the participant.
Add an evaluation
Save and set the status of the item (try all statuses)
When/if you set the status to complete, verify that the item disappears from the list of pending evaluations.
Verify that the evaluated items have the selected status.
Enable the Evaluations tool in My Workspace
Verify that the tool displays all pending evaluations for which you are a desugnated evaluator in all site
Verify that yout can successfully complete evaluations from the aggregated tool
Verify that the name of the matrix cell owner does not display in the "manage evaluations" view or cell view when you do not have the  "Can view / access user list and cell owner" permission.

Instructions:

Replace "Tester Name" in the column header with your name.  When you have successfully completed the action under multiple conditions, put an  in the appropriate cell below.  If you are unable to complete the action due to a  bug, create a Jira and then add a  to the cell as well the Jira ID.

Working with Assignments: Instructors

Action	Tester Name	Sakai QA Version	Comments
Add instructor to portfolio site if s/he is not already in the site	Lynn on QA3-US	2.7.0-b01
Ensure that the course site is associated with Matrices in one or more portfolio site	Lynn on QA3-US	2.7.0-b01
In the course site, open assignments	Lynn on QA3-US	2.7.0-b01
Add and post new assignment (try this with different parameters each time)	Lynn on QA3-US	2.7.0-b01
Once the assignment is saved, use the Create/Edit Matrix Links link in the assignment list to link the assignment to one or more cells in one or more matrices in sites with which the course site is associated.	Lynn on QA3-US	2.7.0-b01
Verify that clicking the "i" icon in a cell displays cell guidance in thickbox.	Lynn on QA3-US	2.7.0-b01
Verify you can not link to matrix cells in unassociated sites (including the originating site).	Lynn on QA3-US	2.7.0-b01
Verify you can not link to matrix cells in unpublished matrices.	Lynn on QA3-US	2.7.0-b01
Verify that information about the saved links displays inline in the assignment.	Lynn on QA3-US	2.7.0-b01	yes, in student view.  It's supposed to display in "view assignment"view for instructor,  but SAK-17733 prevents this read-only view from displaying
Attempt to modify and delete links before students make submissions. This should be possible	Lynn on QA3-US	2.7.0-b01
Attempt to modify and delete links after students make submissions.  Verify that links can be added, but not deleted.	Lynn on QA3-US	2.7.0-b01
Return assignment to student and allow resubmission.  Verify that cell dislays updated submissions	Lynn on QA3-US	2.7.0-b01
Delete linked assignments without submissions.	Lynn on QA3-US	2.7.0-b01	SAK-17854: Incorrect submission count when deleting assignments
Delete linked assignments with submissions.	Lynn on QA3-US	2.7.0-b01	SAK-17854:Incorrect submission count when deleting assignments

Working with Assignments: Students

Action	Tester Name	Sakai QA Version	Comments
Add student to associated portfolio site if not already a member	Lynn on QA3-US	2.7.0-b01
Verify that the Create/Edit Matrix Links is not visible in Assignment list	Lynn on QA3-US	2.7.0-b01
Open an assignment that has been linked to cells in one or more matrices	Lynn on QA3-US	2.7.0-b01
Verify that information about the links is displayed (in a user friendly form)  inline in the assignment	Lynn on QA3-US	2.7.0-b01
Upload or enter data and Save Draft.  Verify that submission is not visible in corresponding matrix cell.	Lynn on QA3-US	2.7.0-b01
Submit assignment (try inline, attachment, both)	Lynn on QA3-US	2.7.0-b01
Open matrices and view cells containing submitted linked assignments	Lynn on QA3-US	2.7.0-b01
Verify that assignment shows up in linked cells.  For iteration 3, the assignment will appear in linked cells regardless of the cells' status.	Lynn on QA3-US	2.7.0-b01	SAK-17853: Unsubmitted linked assignments display in matrix if instructor clicks "grade" or "view submissions" link in Assignments
Verify that you can open the assignment and see full instructions as well as your submission.	Lynn on QA3-US	2.7.0-b01
Resubmit a returned  assignment (that has been linked to matrix cells).  Verify that most recent submission shows up in linked cells.	Lynn on QA3-US	2.7.0-b01

Viewing Linked Assignments in Matrices: (For anyone who can see participant submissions)

Actions	Tester Name	Sakai QA Version	Comments
Verify that submitted assignments appear in linked cells.	Lynn on QA3-US	2.7.0-b01
Verify that the assignment icon and link does not appear in linked cells for participants who have not yet submitted the assignmnent.	Lynn on QA3-US	2.7.0-b01	SAK-17853: Unsubmitted linked assignments display in matrix if instructor clicks "grade" or "view submissions" link in Assignments
Verify that portfolio site members are not required to be member of associated sites in order to see linked assignments.	Lynn on QA3-US	2.7.0-b01
Open the assignment.	Lynn on QA3-US	2.7.0-b01
Verify that you can see both the instructions and student submission.	Lynn on QA3-US	2.7.0-b01
Verify that you have no unexpected permissions (if you are not a member of the associated course, you should only be able to view assignment and submission).	Lynn on QA3-US	2.7.0-b01

Instructions:

Replace "Tester Name" in the column header with your name.  When you have successfully completed the action under multiple conditions, put an  in the appropriate cell below.  If you are unable to complete the action due to a  bug, create a Jira and then add a  to the cell as well the Jira ID.

Authoring Wizards:  Coordinators

Indiana University does not currently use Sequential (step-by-step) wizards.  Please concentrate on Hierarchical (Outline) wizards

Actions	Tester name	Sakai QA Version	Comments
Add a Step-by-Step and Outline wizard	Brian on QA7	2.7.0-b04
Import a Step-by-Step and Outline wizard	Brian on QA7	2.7.0-b04	Bug report on import. SAK-18181
Verify appropriate permissions for each role and that permissions are working properly	Brian on QA7	2.7.0-b04
Export a Step-by-Step and Outline wizard	Brian on QA7	2.7.0-b04
Import 2.5 and a 2.6 Step-by-Step and Outline wizard	Brian on QA7	2.7.0-b04
Delete a Step-by-Step and Outline wizard	Brian on QA7	2.7.0-b04
Edit Wizard Properties	Brian on QA7	2.7.0-b04
Add/edit  description	Brian on QA7	2.7.0-b04
Add and select a new CSS	Brian on QA7	2.7.0-b04
Add categories and pages (Outline wizard)	Brian on QA7	2.7.0-b04
Move categories and pages (Outline wizard)	Brian on QA7	2.7.0-b04
Edit categories and pages (Outline wizard)	Brian on QA7	2.7.0-b04
Delete categories and pages from an unpublished wizard (should not be able to delete from published wizard) (Outline wizard)	Brian on QA7	2.7.0-b04
Edit main wizard page	Brian on QA7	2.7.0-b04
Select both published and unpublished forms	Brian on QA7	2.7.0-b04
Verify that all coordinators can see selected unpublished forms	Brian on QA7	2.7.0-b04	Only the owner can see unpublished forms. No other coordinator has access SAK-18197
Select evaluators	Brian on QA7	2.7.0-b04
Edit  subpages	Brian on QA7	2.7.0-b04
Add/edit/delete description	Brian on QA7	2.7.0-b04
Add custom style (CSS)	Brian on QA7	2.7.0-b04
Enable/disable participant attachments	Brian on QA7	2.7.0-b04
Add/edit/delete each type of guidance.  Verify that guidance labels do not display when field is empty.	Brian on QA7	2.7.0-b04	Two cosmetic issues: When guidance is deleted, the label still displays The title of the guidance jumps above the title bar SAK-18198
Select evaluators	Brian on QA7	2.7.0-b04
Preview a Step-by-Step  and Outline wizard and add test data	Brian on QA7	2.7.0-b04
Publish a Step-by-Step and Outline wizard and verify that test data is deleted	Brian on QA7	2.7.0-b04
Edit Page (these steps only available after current site has been associated with a matrix in the same site or with other portfolio sites)	Brian on QA7	2.7.0-b04
Create link to matrix cells in associated matrices	Brian on QA7	2.7.0-b04
Edit links	Brian on QA7	2.7.0-b04
Delete links	Brian on QA7	2.7.0-b04
Edit or delete links after work has been submitted.			Awaiting participant testing
Test and verify all general and item feedback settings.			Awaiting participant testing
Change the status of a page for the current user (test all status settings)	Brian on QA7	2.7.0-b04
Change the status of a page for a specific group (test all status settings and verify that change affects all members of the group)	Brian on QA7	2.7.0-b04	Groups not appearing in drop down. SAK-18215
Change the status of a page for entire site (test all status settings and verify that change affects all users in the site)	Brian on QA7	2.7.0-b04

Using Wizards: Participants

Actions	Tester Name	Sakai QA Version	Comments
Working with Forms
Add	Brian on QA7	2.7.0-b04
Select existing	Brian on QA7	2.7.0-b04
Edit	Brian on QA7	2.7.0-b04
Delete	Brian on QA7	2.7.0-b04
View a saved form by clicking on its name.
Working with Attachments (Items)
Add
From local storage device
Attach a copy from resources
Remove
Verify that attachment widget is suppressed when "disable participant attachments" is checked in Page Settings.
Working with Reflections
Complete
Edit
Remove
View by clicking on reflection form title.
Submit for Evaluation
Verify that pages progress properly from ready to pending status
Verify that button does not appear if no evaluation form is selected for page
Verify that pages have the appropriate status
Verify that you cannot edit pending and locked pages
Verify that you are notified via email when new feedback is added to he Wizard.
Verify that you are notified via email when a new evaluation is added to he Wizard.

Working with Wizards: Reviewers

Working with Wizards: Evaluators

Related Links

• QA Testers Wall
• QA work flow for QA acceptance testing
• Test Conditions and Scripts
• OSP 2.7 Code Review
• Subversion repository for QA 2.7 Test Scripts: https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/
• Scripts for Testing IU Enhancements

QA Testing from previous releases

• Portfolio 2.6 changes (see Improving Portfolio User Experience)
• 2.6 Selenium Scripts (OSP)
• Functional Test Cases for OSP 2.5
• 2.5 QA Testing
• System Test Scripts
• OSP 2.4 Test Cases