- This line was added.
- This line was removed.
- Formatting was changed.
QA stands for Quality Assurance. In this context the QA mission is primarily to uncover and report software bugs, to verify fixes to software bugs, and to test for regressions. Regressions are features that worked in previous versions, but are broken in the current version.
Pre-requisites for participating
Required - a Jira account is required. Sign up at https://jira.sakaiproject.org
Required - Join Sakai QA email group (firstname.lastname@example.org) or Sakai Dev group (email@example.com) to keep up to date with the latest announcements and ways to help with the testing effort.
Optional - Jira QA group - special permissions - You do not need special permissions to test fixes and to comment on Jira issues. There is a "Tested" button that indicates successful testing of a Jira issue. This button is only available if you are a member of the Jira QA group. This permission is granted to experienced testers. Contact firstname.lastname@example.org for more information.
Hints - Our primary tools are Jira, Google docs spreadsheets, and QA servers. Learning how to use Jira and what the fields represent is well worth the time.
Ways to Participate
When to Participate
How to Participate
Creating Accounts to use for Testing
Testing either takes place on a nightly server, usually trunk, , or on a QA server that is announced when community testing for major or maintenance releases is scheduled. Please be aware that nightly servers are refreshed daily, at midnight Eastern time (-4 or -5 GMT). Please plan your testing appropriately.
Jira bugs which have a resolution of "Fixed" and a status of "Resolved" are ready for testing. If testing is successful click the Tested button. The status will change to Verified. If you don't see a Tested button, even with the correct status and resolution, you may not have permissions in Jira to see the button. Contact email@example.com to request this permission.
In your comments make sure to include which OS/browsers you tested and the Revision number of trunk. The Revision number is important because trunk changes frequently. Look at the bottom of your browser window for the Revision number.
Notes: Some issues may require testing in more than one OS /Browser combination.
Special Case: Trunk vs Branch
Trunk has the most recent source code, and is constantly changing as developers contribute fixes and new features. At some point, when we are getting ready to release Sakai as a community-supported version (aka general availability or production release), we create a branch. A branch is a copy of trunk at a certain point in time, a snapshot. By doing this, we can exercise more control over what goes into a release. We name a branch to correspond with the release. For Sakai 2.9 release we have a 2.9 branch, which we sometimes also refer to as 2.9.x. For Sakai 10 release we have a 10 branch, also called 10.x . Features and fixes almost always start in trunk (more than 99% of the time. Exceptions are rare.) We test features in trunk and then when they are verified to work, if they should be included in a branch, we merge them into the branch and re-test the fixes, since the environment between trunk and the branches can diverge significantly. A fix that works in trunk, may not work in the branch (although in most cases it does.) Since we do not have enough QA resources to retest every fix, we focus on getting at least all the blocker priority issues retested.
We include in JIRA a status that indicates the state of the ticket with respect to the branch. For example: 10 status. When it is set to Resolved it means that the fix has been merged into the Sakai 10.x branch. Verified means that it has been tested after merging into 10.x . This can be confusing, until you get used to it, because it means that there are at least two fields on a JIRA ticket that have the potential to be set as Resolved or Verified, the main Status of the ticket (see image above), and the branch status. There are only 2 versions of Sakai supported at any one time. Currently that is 10 and 2.9, therefore we have a 10 status field and 2.9 status field. When Sakai 11 is released, we will drop support for 2.9 and there will be an 11 status field and a 10 status field.
As a QA tester, once you understand what the fields mean, it is okay to set the status yourself to Verified, on both the overall JIRA and for the branch.
A bug that is supposed to be fixed, isn't
If you are verifying a fix and it fails then reopen the issue by clicking the Reopen button. This indicates to the developer that there is more work to be done. If you are not sure if the issue should be reopened then simply leave a comment. Make sure to include the exact steps of how it failed, what server you were on, the revision number if testing was done on trunk, browser/os, etc.
Testing new features is essentially the same as testing for bug fixes. You still want to click the Tested button to move the issue to Verified status if all the testing passes and Reopen or add comments if you find bugs. Sometimes there are debates about new features and what constitutes a "bug". Those debates happen in the Jira comments and sometimes are brought to other groups like the Sakai Core team for broader discussion, as appropriate/relevant.
Regression testing is detailed testing of features to look for "regressions". Regressions are features that worked in previous versions of Sakai, but now are broken in the current version. Regression testing is detailed testing intended to uncover new regressions.
The most up-to-date list of tools for which we have Regression scripts (steps for testing features manually) is kept in the Tools Google doc.
Release testing - QA Test Hubs
Our goal is to have testing happen as much as possible throughout the Sakai version lifecycle (verifying bug fixes in trunk, testing new features, and regression testing the latest release). However, when we are getting close to having a tagged release (e.g. 2.9.0, 2.9.1, 2.9.2, 2.9.3, etc) we choose specific periods of time to test, with a Test plan and focus. We call these testing efforts "Test Fests". We have multiple Test Fests to get out a release. Unless otherwise specified we have one "Test Fest" per phase of the release. For example, we might have several Beta versions of the software before the Sakai core team feels confident that it is ready for release. Typically, we then have two or three Release Candidates of the software. A Release Candidate is a version of the software, which if it passes QA testing and Developer scrutiny, will become the next official community release. Each of these Beta releases and Release candidates is a phase of our Software Release process.
What makes a Test Fest unique is that we create a Test plan to guide our testing, a defined period of time to complete the testing, and a review process to assess the state of our testing results. Test Fest's typically take a concentrated effort and we like to have as many testers as possible helping in the effort, sometimes with multiple testers focusing on the same functional areas to provide the highest confidence level we can reasonably achieve. Test Fests are almost always carried out on special QA test servers which have been updated with the correct version of the software which needs testing. The Test Plan will indicate which QA servers are available for testing.
Stay tuned for more information.
General Testing Tips