Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Sv translation
languageen

Table of Contents

Purpose

QA stands for Quality Assurance. In this context the QA mission is primarily to uncover and report software bugs, to verify fixes to software bugs, and to test for regressions.  Regressions are features that worked in previous versions, but are broken in the current version.

Pre-requisites for participating

Required - a Jira account is required. Sign up at https://jira.sakaiproject.org

VideoHow to Create a Jira

RequiredJoin Sakai QA email group (sakai-qa@apereo.org) or Sakai Dev group (sakai-dev@apereo.org) to keep up to date with the latest announcements and ways to help with the testing effort.

Optional - Jira QA group - special permissions - You do not need special permissions to test fixes and to comment on Jira issues. There is a "Tested" button that indicates successful testing of a Jira issue. This button is only available if you are a member of the Jira QA group. This permission is granted to experienced testers. Contact neal.caidin@apereo.org for more information.

Hints - Our primary tools are Jira, Google docs spreadsheets, and QA servers. Learning how to use Jira and what the fields represent is well worth the time.

Ways to Participate

  • Verify bugs that have been fixed.
  • Test new features.
  • Regression test existing features. 
  • Create Regression scripts (i.e. step-by-step instructions for performing regression tests.)
  • Release testing - testing alpha, beta, and release candidate versions before software is made generally available (GA aka "General Availability" aka production ready). 
  • Help maintain QA documentation in Confluence. Proof read. Make suggestions. Create short videos. 
  • Ask and answer questions on the email groups. 
  • QA team meetings. Attend the QA meetings to bring up issues that need attention, to help plan for testing for releases, and more.
  • Expert QA knowledge needed - special skills are needed for some types of QA testing.

When to Participate

  • Anytime. We are almost in constant need of QA testing of one form or another. Contact the Sakai Community Coordinator for help getting started.
  • Major Releases - Sakai 2.9, Sakai 10, and Sakai 11 are all examples of major releases. Major releases typically include significant upgrades to existing features and the addition of new features.
  • Maintenance Releases - Sakai 2.9.3 and Sakai 10.5 are examples of maintenance releases. 

How to Participate

  • Email the Sakai Community Coordinator
  • Be on the lookout for email messages asking for QA testing, usually with a link to a QA test plan, which contains information on what needs testing, which server to test on, and how to report results.

 

Creating Accounts to use for Testing

  1. You can create accounts for testing using the "New Account" button on the first page of Sakai (when not logged in).
  2. You can use built-in accounts on most of our QA or nightly servers: instructor, instructor1; and student0001 through student0499 . Password is sakai for demo accounts.

 

Anchor
VerifyingBugs
VerifyingBugs
Verifying bug fixes

Testing either takes place on a nightly server, usually trunk, , or on a QA server that is announced when community testing for major or maintenance releases is scheduled. Please be aware that nightly servers are refreshed daily, at midnight Eastern time (-4 or -5 GMT). Please plan your testing appropriately. 

 

Jira bugs which have a resolution of "Fixed" and a status of "Resolved" are ready for testing. If testing is successful click the Tested button. The status will change to Verified. If you don't see a Tested button, even with the correct status and resolution, you may not have permissions in Jira to see the button. Contact neal.caidin@apereo.org to request this permission.

In your comments make sure to include which OS/browsers you tested and the Revision number of trunk. The Revision number is important because trunk changes frequently. Look at the bottom of your browser window for the Revision number. 

Notes: Some issues may require testing in more than one OS /Browser combination. 

Special Case:

Trunk

Master (formerly k vs Branch

Trunk has the most recent source code, and is constantly changing as developers contribute fixes and new features. At some point, when we are getting ready to release Sakai as a community-supported version (aka general availability or production release), we create a branch. A branch is a copy of trunk at a certain point in time, a snapshot. By doing this, we can exercise more control over what goes into a release. We name a branch to correspond with the release. For Sakai 2.9 release we have a 2.9 branch, which we sometimes also refer to as 2.9.x. For Sakai 10 release we have a 10 branch, also called 10.x .  Features and fixes almost always start in trunk (more than 99% of the time. Exceptions are rare.) We test features in trunk and then when they are verified to work, if they should be included in a branch, we merge them into the branch and re-test the fixes, since the environment between trunk and the branches can diverge significantly. A fix that works in trunk, may not work in the branch (although in most cases it does.) Since we do not have enough QA resources to retest every fix, we focus on getting at least all the blocker priority issues retested.

We include in JIRA a status that indicates the state of the ticket with respect to the branch. For example: 10 status. When it is set to Resolved it means that the fix has been merged into the Sakai 10.x branch. Verified means that it has been tested after merging into 10.x . This can be confusing, until you get used to it, because it means that there are at least two fields on a JIRA ticket that have the potential to be set as Resolved or Verified, the main Status of the ticket (see image above), and the branch status. There are only 2 versions of Sakai supported at any one time. Currently that is 10 and 2.9, therefore we have a 10 status field and 2.9 status field. When Sakai 11 is released, we will drop support for 2.9 and there will be an 11 status field and a 10 status field.

As a QA tester, once you understand what the fields mean, it is okay to set the status yourself to Verified, on both the overall JIRA and for the branch.

 

A bug that is supposed to be fixed, isn't

If you are verifying a fix and it fails then reopen the issue by clicking the Reopen button. This indicates to the developer that there is more work to be done. If you are not sure if the issue should be reopened then simply leave a comment. Make sure to include the exact steps of how it failed, what server you were on, the revision number if testing was done on trunk, browser/os, etc.

Anchor
TestNewFeatures
TestNewFeatures
Test New Features

Testing new features is essentially the same as testing for bug fixes. You still want to click the Tested button to move the issue to Verified status if all the testing passes and Reopen or add comments if you find bugs. Sometimes there are debates about new features and what constitutes a "bug". Those debates happen in the Jira comments and sometimes are brought to other groups like the Sakai Core team for broader discussion, as appropriate/relevant. 

Anchor
RegressionTest
RegressionTest
Regression testing

Regression testing is detailed testing of features to look for "regressions". Regressions are features that worked in previous versions of Sakai, but now are broken in the current version. Regression testing is detailed testing intended to uncover new regressions. 

The most up-to-date list of tools for which we have Regression scripts (steps for testing features manually) is kept in the Tools Google doc

Canonical list of Regression Scripts.

Release testing - QA Test Hubs

Our goal is to have testing happen as much as possible throughout the Sakai version lifecycle (verifying bug fixes in trunk, testing new features, and regression testing the latest release). However, when we are getting close to having a tagged release (e.g. 2.9.0, 2.9.1, 2.9.2, 2.9.3, etc) we choose specific periods of time to test, with a Test plan and focus. We call these testing efforts "Test Fests". We have multiple Test Fests to get out a release. Unless otherwise specified we have one "Test Fest" per phase of the release. For example, we might have several Beta versions of the software before the Sakai core team feels confident that it is ready for release. Typically, we then have two or three Release Candidates of the software. A Release Candidate is a version of the software, which if it passes QA testing and Developer scrutiny, will become the next official community release. Each of these Beta releases and Release candidates is a phase of our Software Release process. 

What makes a Test Fest unique is that we create a Test plan to guide our testing, a defined period of time to complete the testing, and a review process to assess the state of our testing results. Test Fest's typically take a concentrated effort and we like to have as many testers as possible helping in the effort, sometimes with multiple testers focusing on the same functional areas to provide the highest confidence level we can reasonably achieve. Test Fests are almost always carried out on special QA test servers which have been updated with the correct version of the software which needs testing. The Test Plan will indicate which QA servers are available for testing. 

Anchor
QATeamMeetings
QATeamMeetings
QA team meetings

Stay tuned for more information. 

Anchor
ExpertQAKnowledge
ExpertQAKnowledge
Expert QA knowledge

  • Accessibility Testing
  • Localization Testing (different languages)
  • Review of patches and fixes
  • Testing locally
  • Automated testing
  • Security testing (requires special permission)

General Testing Tips

Include Page
General Testing Tips
General Testing Tips

 

Sakai 11 QA Hub

Meeting Logistics 

BigBlueButton hosted by BlindsideNetworks

http://apereo.blindsidenetworks.net/apereo/

Room "Sakai QA"

password apereo

Previous meeting minutes

December 17, 2015

Today's meeting minutes

January 14, 2016

 

...