QA stands for Quality Assurance. In this context the QA mission is primarily to uncover and report software bugs, to verify fixes to software bugs, and to test for regressions. Regressions are features that worked in previous versions, but are broken in the current version.
Pre-requisites for participating
Required - a Jira account is required. Sign up at https://jira.sakaiproject.org
Required - Join Sakai QA email group (firstname.lastname@example.org) or Sakai Dev group (email@example.com) to keep up to date with the latest announcements and ways to help with the testing effort.
Optional - Jira QA group - special permissions - You do not need special permissions to test fixes and to comment on Jira issues. There is a "Tested" button that indicates successful testing of a Jira issue. This button is only available if you are a member of the Jira QA group. This permission is granted to experienced testers. Contact firstname.lastname@example.org for more information.
Hints - Our primary tools are Jira, Google docs spreadsheets, and QA servers. Learning how to use Jira and what the fields represent is well worth the time.
Ways to Participate
- Verify bugs that have been fixed.
- Test new features.
- Regression test existing features.
- Create Regression scripts (i.e. step-by-step instructions for performing regression tests.)
- Release testing - testing alpha, beta, and release candidate versions before software is made generally available (GA aka "General Availability" aka production ready).
- Help maintain QA documentation in Confluence. Proof read. Make suggestions. Create short videos.
- Ask and answer questions on the email groups.
- QA team meetings. Attend the QA meetings to bring up issues that need attention, to help plan for testing for releases, and more.
- Expert QA knowledge needed - special skills are needed for some types of QA testing.
Verifying bug fixes
Most testing should take place on the Trunk QA nightly server. Please be aware that this server is refreshed with the latest version of Sakai every 4 hours, starting at midnight Eastern time (-4 or -5 GMT). Please plan your testing appropriately. It takes about 20 minutes for Sakai to rebuild and deploy. So, for example, between 12:20 pm and 3:59 pm Eastern time is a good time to test.
Jira bugs which have a resolution of "Fixed" and a status of "Resolved" are ready for testing. If testing is successful click the Tested button. The status will change to Verified. If you don't see a Tested button, even with the correct status and resolution, you may not have permissions in Jira to see the button. Contact email@example.com to request this permission.
In your comments make sure to include which OS/browsers you tested and the Revision number of trunk. The Revision number is important because trunk changes frequently. Look at the bottom of your browser window for the Revision number.
Notes: Some issues may require testing in more than one OS /Browser combination.
A bug that is supposed to be fixed, isn't
If you are verifying a fix and it fails then reopen the issue by clicking the Reopen button. This indicates to the developer that there is more work to be done. If you are not sure if the issue should be reopened then simply leave a comment. Make sure to include the exact steps of how it failed, what server you were on, the revision number if testing was done on trunk, browser/os, etc.
Test New Features
Testing new features is essentially the same as testing for bug fixes. You still want to click the Tested button to move the issue to Verified status if all the testing passes and Reopen or add comments if you find bugs. Sometimes there are debates about new features and what constitutes a "bug". Those debates happen in the Jira comments and sometimes are brought to other groups like the CLE Release team or the Samigo team for broader discussion, as appropriate/relevant.
Regression testing is detailed testing of features to look for "regressions". Regressions are features that worked in previous versions of Sakai, but now are broken in the current version. Regression testing is detailed testing intended to uncover new regressions.
The most up-to-date list of tools for which we have Regression scripts (steps for testing features manually) is kept in the Tools Google doc.
Release testing - "Test Fests"
Our goal is to have testing happen as much as possible throughout the Sakai version lifecycle (verifying bug fixes in trunk, testing new features, and regression testing the latest release). However, when we are getting close to having a tagged release (e.g. 2.9.0, 2.9.1, 2.9.2, 2.9.3, etc) we specify specific periods of time to test, with a Test plan and focus. We call these testing efforts "Test Fests". We have multiple Test Fests to get out a release. Unless otherwise specified we have one "Test Fest" per phase of the release. For example, we might have several Beta versions of the software before the CLE release team feels confident that it is ready for release. Typically, we then have two or three Release Candidates of the software. A Release Candidate is a version of the software, which if it passes QA testing and Developer scrutiny, will become the next official community release. Each of these Beta releases and Release candidate version is a phase of our Software Release process.
What makes a Test Fest unique is that we create a Test plan to guide our testing, a defined period of time to complete the testing, and a review process to assess the state of our testing results. Test Fest's typically take a concentrated effort and we like to have as many testers as possible helping in the effort, sometimes with multiple testers focusing on the same functional areas to provide the highest confidence level we can reasonably achieve. Test Fests are almost always carried out on special QA test servers which have been updated with the correct version of the software which needs testing. The Test Plan will indicate which QA servers are available for testing.
QA team meetings
Stay tuned for more information.
Expert QA knowledge
- Accessibility Testing
- Localization Testing (different languages)
- Review of patches and fixes
- Testing locally
- Automated testing
- Security testing (requires special permission)