Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 43 Next »

Purpose

QA stands for Quality Assurance. In this context the QA mission is primarily to uncover and report software bugs, to verify fixes to software bugs, and to test for regressions.  Regressions are features that worked in previous versions, but are broken in the current version.

Pre-requisites for participating

Required - a Jira account is required. Sign up at https://jira.sakaiproject.org

RequiredJoin Sakai QA email group (sakai-qa@collab.sakaiproject.org) or Sakai Dev group (sakai-dev@collab.sakaiproject.org) to keep up to date with the latest announcements and ways to help with the testing effort.

Optional - Jira QA group - special permissions - You do not need special permissions to test fixes and to comment on Jira issues. There is a "Tested" button that indicates successful testing of a Jira issue. This button is only available if you are a member of the Jira QA group. This permission is granted to experienced testers. Contact neal.caidin@apereo.org for more information.

Hints - Our primary tools are Jira, Google docs spreadsheets, and QA servers. Learning how to use Jira and what the fields represent is well worth the time.

Ways to Participate

  • Verify bugs that have been fixed.
  • Test new features.
  • Regression test existing features. 
  • Create Regression scripts (i.e. step-by-step instructions for performing regression tests.)
  • Release testing - testing alpha, beta, and release candidate versions before software is made generally available (GA aka "General Availability" aka production ready). 
  • Help maintain QA documentation in Confluence. Proof read. Make suggestions. Create short videos. 
  • Ask and answer questions on the email groups. 
  • QA team meetings. Attend the QA meetings to bring up issues that need attention, to help plan for testing for releases, and more.
  • Expert QA knowledge needed - special skills are needed for some types of QA testing. 

Verifying bug fixes

Most testing should take place on the Trunk QA nightly server. Please be aware that this server is refreshed with the latest version of Sakai every 4 hours, starting at midnight Eastern time (-4 or -5 GMT). Please plan your testing appropriately. It takes about 20 minutes for Sakai to rebuild and deploy. So, for example, between 12:20 pm and 3:59 pm Eastern time is a good time to test.

Jira bugs which have a resolution of "Fixed" and a status of "Resolved" are ready for testing. If testing is successful click the Tested button. The status will change to Verified. If you don't see a Tested button, even with the correct status and resolution, you may not have permissions in Jira to see the button. Contact neal.caidin@apereo.org to request this permission.

In your comments make sure to include which OS/browsers you tested and the Revision number of trunk. The Revision number is important because trunk changes frequently. Look at the bottom of your browser window for the Revision number. 

Notes: Some issues may require testing in more than one OS /Browser combination. 

A bug that is supposed to be fixed, isn't

If you are verifying a fix and it fails then reopen the issue by clicking the Reopen button. This indicates to the developer that there is more work to be done. If you are not sure if the issue should be reopened then simply leave a comment. Make sure to include the exact steps of how it failed, what server you were on, the revision number if testing was done on trunk, browser/os, etc.

Test New Features

Testing new features is essentially the same as testing for bug fixes. You still want to click the Tested button to move the issue to Verified status if all the testing passes and Reopen or add comments if you find bugs. Sometimes there are debates about new features and what constitutes a "bug". Those debates happen in the Jira comments and sometimes are brought to other groups like the Sakai Core team or the Samigo team for broader discussion, as appropriate/relevant. 

Regression testing

Regression testing is detailed testing of features to look for "regressions". Regressions are features that worked in previous versions of Sakai, but now are broken in the current version. Regression testing is detailed testing intended to uncover new regressions. 

The most up-to-date list of tools for which we have Regression scripts (steps for testing features manually) is kept in the Tools Google doc.

Release testing - "Test Fests"

Our goal is to have testing happen as much as possible throughout the Sakai version lifecycle (verifying bug fixes in trunk, testing new features, and regression testing the latest release). However, when we are getting close to having a tagged release (e.g. 2.9.0, 2.9.1, 2.9.2, 2.9.3, etc) we specify specific periods of time to test, with a Test plan and focus. We call these testing efforts "Test Fests". We have multiple Test Fests to get out a release. Unless otherwise specified we have one "Test Fest" per phase of the release. For example, we might have several Beta versions of the software before the CLE release team feels confident that it is ready for release. Typically, we then have two or three Release Candidates of the software. A Release Candidate is a version of the software, which if it passes QA testing and Developer scrutiny, will become the next official community release. Each of these Beta releases and Release candidates is a phase of our Software Release process. 

What makes a Test Fest unique is that we create a Test plan to guide our testing, a defined period of time to complete the testing, and a review process to assess the state of our testing results. Test Fest's typically take a concentrated effort and we like to have as many testers as possible helping in the effort, sometimes with multiple testers focusing on the same functional areas to provide the highest confidence level we can reasonably achieve. Test Fests are almost always carried out on special QA test servers which have been updated with the correct version of the software which needs testing. The Test Plan will indicate which QA servers are available for testing. 

QA team meetings

Stay tuned for more information. 

Expert QA knowledge

  • Accessibility Testing
  • Localization Testing (different languages)
  • Review of patches and fixes
  • Testing locally
  • Automated testing
  • Security testing (requires special permission)

General Testing Tips

Overview

This section serves to provide general testing tips for web applications. It provides basic tests for different elements in web applications to help users think about different ways to test. It is not meant to provide comprehensive documentation on every way to test.

Categorized tips
Data variations

Variations in data that should be used throughout the application in every location that strings can be entered.

  • Special characters in text ~!@#$%^&*()
  • Non-Latin characters (Cyrillic, Arabic, Japanese, Chinese, Klingon, etc.)
  • Languages from right to left (Arabic, Hebrew) [http://www.lipsum.com is a good site to get sample text for both non-latin characters and languages reading from right to left]
  • Latin characters with diacritical marks
  • Latin characters using ligatures
  • Very long strings
  • Single character strings
  • Scripts/HTML tags
    • <script>alert("xss");</script>
    • Content should be handled gracefully, but not executed when displayed, unless explicitly stated for a feature
Form testing tips
  • Submit with no form values
  • Submit without required fields filled out
  • Submit with improperly formed values (i.e. an improperly formed email in an email address field)
  • Submit with Data variations (see Data Variations section)
Login screen testing tips
  • Submit with no values
  • Valid username, invalid password
  • Valid username, valid password
  • Invalid username
  • Valid username, no password
Drop Down menus
  • Drop down is properly sized for menu options
  • Default menu option is displayed in drop down on page load
  • Selected menu option is displayed in drop down after clicking off menu
  • Drop down menu can be navigated through with keyboard and mouse
  • Drop down menu appropriately resizes vertically on small screen resolutions
  • Entering a character selects the first menu option starting with that character
Combo box
  • Combo box is properly sized for menu options
  • Default menu option is displayed in drop down on page load
  • Can select single item
  • Can select multiple items in list
  • Can select non-contiguous items
  • Drop down menu can be navigated through with keyboard and mouse
  • Drop down menu appropriately resizes vertically on small screen resolutions
  • Entering a character selects the first menu option starting with that character

 

  • No labels