QA stands for Quality Assurance. In this context the QA mission is primarily to uncover and report software bugs, to verify software bugs that have been fixed, and to test new features for regressions (new bugs inadvertently introduced when adding a new feature) for the Sakai CLE software.
Pre-requisites for participating
Required - a Jira account is required. Sign up at https://jira.sakaiproject.org
Required - Join Sakai QA email group (email@example.com) or Sakai Dev group (firstname.lastname@example.org) to keep up to date with the latest announcements and ways to help with the testing effort.
Optional - Jira QA group - special permissions - You do not need special permissions to test fixes and to comment on Jira issues. There is a "Tested" button that indicates successful testing of a Jira issue. This button is only available if you are a member of the Jira QA group. This permission is granted to experienced testers. Contact email@example.com for more information.
Hints - Our primary tools are Jira, Google docs spreadsheets, and QA servers. Learning how to use Jira and what the fields represent is well worth the time.
Ways to Participate
- Verify bugs that have been fixed.
- Test new features.
- Regression test existing features.
- Create Regression scripts (i.e. step-by-step instructions for performing regression tests.)
- Release testing - testing alpha, beta, and release candidate versions before software is made generally available (GA aka production).
- Help maintain QA documentation in Confluence. Proof read. Make suggestions. Create short videos.
- Ask and answer questions on the email groups.
- QA team meetings. Attend the QA to bring up issues that need attention, to help plan for testing for releases, and more.
- Expert QA knowledge needed - special skills are needed for some types of QA testing.
Verifying bug fixes
Most testing should take place on the Trunk QA nightly server. Please be aware that this server is refreshed with the latest version of Sakai every 4 hours, starting at midnight Eastern time (-4 or -5 GMT). Please plan your testing appropriately. It takes about 20 minutes for Sakai to rebuild and deploy. So, for example, between 12:20 pm and 3:59 pm Eastern time is a good time to test.
Jira bugs which have a resolution of "Fixed" and a status of "Resolved" are ready for testing. If testing is successful click the Tested button. The status will change to Verified. If you don't see a Tested button, even with the correct status and resolution, you may not have permissions in Jira to see the button. Contact firstname.lastname@example.org to request this permission.
In your comments make sure to include which OS/browsers you tested and the Revision number of trunk. The Revision number is important because trunk changes frequently. Look at the bottom of your browser window for the Revision number.
Notes: Some issues may require testing in more than one OS /Browser combination.
A bug that is supposed to be fixed, isn't
If you are verifying a fix and it fails then reopen the issue by clicking the Reopen button. This indicates to the developer that there is more work to be done. If you are not sure if the issue should be reopened then simply leave a comment. Make sure to include the exact steps of how it failed, what server you were on, the revision number if testing was done on trunk, browser/os, etc.
Test New Features
Testing new features is essentially the same as testing for bug fixes. You still want to click the Tested button to move the issue to Verified status if all the testing passes and Reopen or add comments if you find bugs. Sometimes there are debates about new features and what constitutes a "bug". Those debates happen in the Jira comments and sometimes are brought to other groups like the CLE Release team or the Samigo team for broader discussion, as appropriate/relevant.
Regression testing is detailed testing of features to look for "regressions". Regressions are features that worked in previous versions of Sakai, but now are broken in the current version. Regression testing is detailed testing intended to uncover new regressions.
The most up-to-date list of tools for which we have Regression scripts (steps for testing features manually) is kept in the Tools Google doc.
When we are getting close to having a "tagged" community release, we typically have a "Test Fest". We will designate specific Jiras which need re-testing in 2.9.x even though they have already been tested on the Trunk QA server. We will have specific QA servers available and the details will be communicated to the Community through the email groups, especially Sakai QA and Dev email groups.
QA team meetings
Stay tuned for more information.
Expert QA knowledge
- Accessibility Testing
- Localization Testing (different languages)
- Review of patches and fixes
- Testing locally
- Automated testing
- Security testing (requires special permission)