Panel Session: Sakai QA Process

Session 074

Panel members: Peter Knoop (UM), Stephen Marquard (UCT), Clay Fenlason (BU), Seth Theriault (Columbia), Victor Maijer (UVA)

Thursday
1:30 pm-3:30 pm
Room: Rm404

Session Abstract

Discussion of overall Sakai QA process with panel of active QA WG members.

See also the Sakai WG: QA space for more information on Sakai QA.

Presentation Materials

Peter Knoop presented some slides showing inter alia QA figures for the 2.1 release cycle. Participation came from:

Individuals: 52
Institutions: 27
Countries: 6 (Portugal, South Africa, Sweden, The Netherlands, United Kingdom, United States)

	Name	Size	Creator (Last Modifier)	Creation Date	Last Mod Date	Comment
	QA_Panel.mp3	35.46 MB	Peter A. Knoop	Dec 13, 2005	Dec 13, 2005	First hour only of Panel because iPod battery died.

Introductory comments from the panel

Clay (release documentation):

• Got involved as non-techie - not a programmer/developer nor sysadmin. Wanted to understand system better: volunteering for QA a good idea to do that - involved with deployment QA from 1.5
• Documentation was a good way to contribute - "working out confusion in a public way" - docs were a deliverable of the deployment QA process.
• Confluence has become a staging area for docs - "documentation trunk". Checked into SVN once firmed up.

Seth (install / deployment QA):

• "Smaller half" of the QA process i.t.o. the number of individuals involved. Goal is to take install documentation and try it out on different systems. Benefits from a variety of different systems (e.g. Columbia: Solaris, Oracle backend, BU: Linux + mysql, Rutgers: Solaris + mysql, MacOS X with mysql)
• Make sure that it's deployable on all these combinations of platforms.
• Being part of documentation effort - not always obvious how to do things. Sysadmin people work from checklists.
• Adding details on version upgrades.

Victor (issue verification, regression testing):

• Started after Baltimore
• Sysadmin with Blackboard background
• Quite a learning curve - wanted to help but processes weren't clear.
• Started FAQ doc.
• If we want to attract new people, we need to guide them
• Regression testing is a lot of work!
• More communication required to recruit new QA people.

Stephen (Test Conditions):

• UCT team (5 people) joined QA shortly before 2.0 release and the Baltimore conference.
• A very effective way of learning about the Sakai tools, code and community.
• Helped us significantly in planning for our own deployment. A way of reducing risk, as we have a better understanding of the readiness of Sakai for production and any areas to watch out for.
• Involved in the effort to create Test Condition matrices for all Sakai tools. These provide a test plan which QA people can work through to test as much functionality as possible in each tool.
• TCs are not yet complete - the goal is to have a complete set of TCs which can be applied methodically in a future release cycle, e.g. 2.2.

Peter (JIRA manager):

• Just finished 2.1 QA cycle
• 2.1.1 release QA cycle coming up in Jan.
• Biggest struggle is getting people organized and together.
• FAQ posted on Confluence site.
• Please volunteer!

Discussion notes

There was wide-ranging discussion with some concrete suggestions for improving processes and new approaches.

• Is there a timeline for automated testing (rather than working through TCs)? Short answer: no, lots of options (including test harnesses, unit testing, web services), dependent on people contributing work.
• Want to applaud Clay for the release documentation. Helpful for all documentation would be to identify versions of Sakai that it refers to (e.g. the doc "How to configure Sakai" from May 05 is now out of date).
• There are some ongoing discussions about how to handle docs in general - up for discussion. Need to involve developers on that. Code changes are rapid which obsoletes documentation quickly - process needs to evolve.
• People are at different stages in release cycles (using different versions of Sakai), and need docs related to earlier versions. Use SVN to do versioning to manage doc cycle?
• Lots of discussion about frozen/unfrozen docs and how its tied to releases. All agreed: versioning is important. Releasing docs as MS Word docs is unhelpful.
• Documentation is fundamental for community source processes. A lack of documentation can inhibit potential developers from contributing.
• QA exposed the documentation issue - QA people need authoritative references about functionality when filing or verifying bug reports.
• Some institutions have their own documentation ("secret sauce") - people think locally but teams should think of themselves as part of the community, and publish internal documentation (for example about configuration and processes) wherever possible.
• One option for doing this is to put up local institutional docs on local wikis, which are publicly accessible. There are already a number of these wikis, e.g. UCT (for SA Sakai community).
• When should adopters start thinking about production? QA seems conspicuously absence in a lot of discussion. The dev team is often running - "we need this now!". Seems like the wrong things are driving the train. Some past conversations which institutions had with BB executive leadership revolved around QA: "biggest problem is instability of releases" - BB started with 45 developers, 6 QA, and inverted the ratio to 30 developers, 135 involved in software engineering on QA.
• Michigan is taking a more cautious approach to introducing new releases, and is now doing performance testing first.
• As campuses become more engaged, engagement will get more people volunteering to have better QA. The community will demand more quality.
• What value does the community put on QA?
• An FTE is good, but community needs ongoing support
• Carol Dippel moved mountains as QA lead - significant progress has been made.
• Need discussion in community
• Will release processes slow down? Maybe, but maybe not.
• QA is not sexy! The BB example (relative priority of QA) is very important. If QA is not taken seriously enough, it could lead to a backlash and collateral damage.
• As more people make commitments to Sakai, QA becomes a bigger need.
• The key ingredient is stability.
• How insulated are Sakai teams from users? Where users report problems directly, QA is taken seriously. When good QA is done, there are fewer support issues.** * Who will be doing QA?
• There is now a maintenance branch for 2.1.
• QA should get be involved as early as possible - for example Indiana (Kristol) would welcome QA involvement in Message Centres prior to code completion, e.g. to create a test plan.
• Communication across / between development stages or processes is the key (e.g. QA team working with developers)
• Does QA include usability issues? Yes, though there is also a separate UI DG focusing on that area.
• VT has assigned 1 FTE to QA processes.
• QA is important, but overall Quality as an approach is also important. QA & UI needs to be in closer contact.
• Unit testing an important part of ensuring code quality
• UCD is also important, e.g. Indiana worked closely with faculty in developing Message Centres - all contributes to improving quality and reducing support issues.
• The warm body factor is important - need to get more people involved. To do that, we need a more flexible schedule which allows people to contribute when they have time (related to institutional priorities at different times)
• More time for QA in the release cycle.
• To take forward the discussion about processes, release cycles, integration between dev and QA teams etc., join the new Community Practices WG (which emerged following the the Integration DG Position Paper)
• What can be done with automated testing?
• Is verification of resolved issues a good use of resources (people's time)?
• Logistical issues - what about the impact of introducing new dbs, e.g. MSSQL ?
• What QA models do other projects have, e.g. Apache ? It's easier to test server products than products with extensive user interaction.
• Most OSS projects fix bugs when reported, and close them when fixed.
• What are best practices for OSS projects?
• Dispersed documentation problem - difficult for implementers to RTFM when there are too many Ms (which M?)
• Threshold to enter Sakai is higher than other projects
• What role for automated testing? Unit tests can be created and run against various dbs.
• Example of gated: hired short-term contractors, people looking for some experience to write regression tests
• Need to reach a certain coverage point of unit testing, after which it will be easier to maintain.
• Getting people to create unit tests, etc.? New entrants to Sakai, or should Foundation pay for people to do this?
• Some entry points (such as unit tests) are not appealing to new entrants - hard to see immediate value of work.
• Provisional > non-provisional tool transition should require unit tests
• Code coverage tools - could show % of APIs which have been unit-tested for example.
• Allow people to assess the risk of using a tool, e.g. "this tool has 90% unit test coverage"
• Balance agility and quality: full disclosure approach, describing extent and nature of QA applied or done at a modular level (tools, services, etc.)
• Everyone looks to institutions similar to their own to see what they're doing - provides a measure of "community confidence"
• We are working out how Community Source is different to OSS - QA process could be one differentiator
• Profiling tools - could be used to verify most-used cost first, or to identify the regions of code called most frequently, and would allow a way to prioritize testing (and need for unit tests).
• Build up threat models / risk models (what are the worst-case failures?)
• Load testing can show performance hotspots
• Could bug marathons, spotlights, sprints - e.g. squash XX bugs over 24 hours. It's easier to commit to some short-term spurt of effort than over the long haul.
• Work around commitment phobia - signing up to a release QA cycle is a significant commitment
• Load-testing: UM could contribute pseudo-scripts which could be used as a basis for JMeter tests
• Lead to a comparable performance metric: "Sakaimark" to establish expected performance on different hardware platforms
• Is it worth doing automated web UI testing? Possibly for automated validation of output (CSS, HTML, javascript), but otherwise these tests are very brittle, and constantly broken by tool changes, so it's better to focus automated testing on the service layer.
• Can we reduce the 3 db test platforms to 1? At the very least, HSQL should be dropped as a QA test platform.
• Can we create a portable QA instance, i.e. which people can use to run QA tests on locally? Important for bandwidth concerns (e.g. QA people in countries from which access to US is still relatively slow) and timing and flexibility - get a QA instance up faster than the MIT instances.
• Could be done as a live CD or live DVD - boot it up, apply latest source, standardized build and runtime config.