Manual Testing of Terracotta Clustering with the Wiley Worksite Setup Tool

Slightly Out of Date This page is just slightly out of date. Please refer to the System Administrator Guide for a more up to date version of most of this material. The only parts that do not appear in the System Administrator Guide, such as HA Proxy configuration, may be used from this page if desired.

Introduction

The development of adding Sakai Session Clustering via Terracotta High Level Design is almost complete (SAK-13324). Testing was conducted to verify the progress made so far has been successful. This test cases are described below.

Wiley Session Clustering Distro (Custom Worksite Setup Tool)

¹ Server was shutdown using the Tomcat shutdown.sh script
² Server was shutdown using the Unix kill -9 pid command
³ Either no JAVA_OPTS flags were used, or the sakai.cluster.terracotta flag was set to false
⁴ The sakai.cluster.terracotta JAVA_OPT was used and set to true along with the other configuration documented below
All Clustering Off cases were expected to fail

Session Clustering Distro (Normal Worksite Setup Tool)

¹ Server was shutdown using the Tomcat shutdown.sh script
² Server was shutdown using the Unix kill -9 pid command
³ Either no JAVA_OPTS flags were used, or the sakai.cluster.terracotta flag was set to false
⁴ The sakai.cluster.terracotta JAVA_OPT was used and set to true along with the other configuration documented below
All Clustering Off cases were expected to fail
Both Clustering On, Workstite Creation cases were expected to fail

Configuration

The following sections describe how to configure an environment to work with Sakai and Terracotta Clustering (as developed under SAK-13324).

Sakai Configuration

If you use the Wiley Session Clustering Distro you must configure the following in your sakai.properties file.

clusterableTools@org.sakaiproject.tool.api.SessionManager=sakai.login,sakai.sitesetup,
sakai.siteinfo,sakai.sitebrowser,sakai.membership

The above should all be on one line, but has been wrapped for legibility.

Additional configuration for the Custom Worksite Setup tool is required. This documentation can be found on the Wiley Worksite Setup Tool Configuration page.

HA Proxy Configuration

1. Obtain HAProxy from the website http://haproxy.1wt.eu/
2. Use the following HAProxy configuration (loosely based on this page and this page)

global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 4096

defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    redispatch
    maxconn 2000
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000

listen webfarm 192.168.1.2:8888
    mode http
    balance roundrobin
    cookie SERVERID insert nocache indirect
    option forwardfor
    option httpchk HEAD /check.txt HTTP/1.0

    # Stats
    stats enable
    stats auth admin:password

    # first Node
    server SBNode1 192.168.1.2:8080 cookie Server1 check
    # second Node
    server SBNode2 192.168.1.2:9080 cookie Server2 check

Notes
1: In the above configuration file 192.168.1.2 must be set to the appropriate IP address for the target environment. Also the port numbers (8888, 8080, and 9080) will vary by environment.
2: A file check.txt must be placed in the Tomcat ROOT web application directory. This file can be a simple text file that is only a few bytes long. The contents of the file are not important.

Tomcat Configuration

There are two different parts of Tomcat that must be configured. The server.xml file found in the conf/ directory must be changed to enable some additional logging. Second, some JAVA_OPTS must be passed to the Tomcat startup sequence.

server.xml

To help determine which server is being hit in a cluster, it's important to enable access logging. This is relatively easy to do, because the configuration is already written, it just needs to be uncommented. Find the following section and remove the XML comment lines above and below this section.

<Valve className="org.apache.catalina.valves.AccessLogValve"
                 directory="logs"  prefix="localhost_access_log." suffix=".txt"
                 pattern="common" resolveHosts="false"/>

With this change enabled, a new log file of the format, localhost_access_log.YYYY-MM-DD.txt should be created in the Tomcat logs directory. Watching this log can help determine whether a particular Tomcat server is being hit or not.

JAVA_OPTS

There are several options that must be passed to Tomcat to enable Terracotta clustering.

• -Dsakai.cluster.terracotta=true
  Indicates that Terracotta clustering should be enabled. Initially this will cause the Sakai Component ClassLoaders to register themselves with Terracotta. After Sakai is running, it will cause the SessionManager to place any NON-clustered tool data in a special non-clustered data structure, while all cluster-enabled tools will be stored in the normal data structure, that is then clustered by Terracotta.
• -Dtc.install-root=/path/to/terracotta/home
  This option must be set to the home directory of the Terracotta distribution. It will find different libraries and modules it needs to run in this directory.
• -Dtc.config=/path/to/tc-config.xml
  This option must be set to the fully qualified path to the Terracotta config file. This will typically be stored in the sakai directory inside of Tomcat.
• -Xbootclasspath/p:$TC_HOME/lib/dso-boot/dso-boot-hotspot_linux_150_14.jar
  This option must be set to the custom boot jar in the Terracotta distribution. This boot jar can be created by running the following Terracotta command in the same environment where you start Tomcat (e.g., same JAVA_HOME environment variable).

  $TC_HOME/bin/make-boot-jar.sh -f /path/to/tc-config.xml

  Once the boot jar has been created, then the previous -Xbootclasspath/p: JAVA_OPT should be set to that file.

Terracotta Configuration

Most of the Terracotta configuration is done with the tc-config.xml file provided in the Sakai terracotta-config module.

Install Terracotta

The first version of Terracotta that was used successfully was version 2.6.4. Other versions may work, but 2.6.4 is known to work with the Sakai code tested at the beginning of September 2008. Terracotta 2.6.4 may be obtained from the www.terracotta.org website under downloads. Once the terracotta-generic-2.6.4.tar.gz package is obtained, simply unarchive it where you would like it to reside on your system. You must create the boot jar file as described in the previous section (see the make-boot-jar.sh command).

Install collection-tims

An additional Terracotta library is needed to successfully run the Sakai Terracotta code. This library is call the Terracotta Integration Module for auto-locked Collections or tim-collections for short. Tim-collections can be downloaded from the Terracotta Forge web site. Version 2.2.2 was known to work with the Sakai code tested at the beginning of September 2008. Once downloaded all of the jar files contained in the download should be copied to the modules subdirectory in the Terracotta installation directory.

$ ls $TC_HOME/modules
org                                   tim-synchronizedmap-2.2.2.jar
readme.html                           tim-synchronizedset-2.2.2.jar
tim-collections-2.2.2.jar             tim-synchronizedsortedmap-2.2.2.jar
tim-hashtable-2.2.2.jar               tim-synchronizedsortedset-2.2.2.jar
tim-synchronizedcollection-2.2.2.jar  tim-vector-2.2.2.jar

Starting Terracotta

To start terracotta, simply run the start-tc-server.sh command in the bin directory of the Terracotta installation. You must pass it a -f /path/to/tc-config.xml command line argument pointing to the same Terracotta configuration file used by Tomcat (see the Tomcat configuration section above). You may optionally set a TC_HOME environment variable to the root of the Terracotta installation directory before starting Terracotta.

Source Code to Build From

When this testing was first conducted (early September 2008) the changes to enable Terracotta clustering in Sakai, were not checked into any mainline Sakai branch or trunk. Instead they exist only in institutional branches. The following institutional distributions may be used to experiment with Terracotta clustering in Sakai.

Session Clustering

This distribution represents the original development environment for the SAK-13324 work. This contains all the framework changes to enable allowing tools to be cluster enabled. It does not, however, enable any Sakai tool, other then login itself.

Wiley Session Clustering

This distribution represents the same code as the previous Session Clustering distribution with the addition of a branch of the site-manage module. This branch, introduces customizations to the Worksite Setup tool. This specific version of Worksite Setup was customized to enable clustering for the tool.

Order of Application Startup

Although many variations potentially exist with regards to the order the applications may be started, the testing described on this page was done with the following order. It is vital, however, that with order is used, the Terracotta server be started before any Sakai Tomcat instance.

1. Start Terracotta (Terracotta start-tc-server.sh command)
2. Start each Sakai Tomcat instance that will participate in the Cluster (Tomcat startup.sh command)
3. Start HAProxy (HAProxy haproxy command)

Test Case

Two tests were run. The first test was designed to verify whether a user must login again on failover, or whether they can continue to use the system without relogging into Sakai. The second test was designed to verify a users progress through a workflow of steps in a particular tool (Worksite Setup) was maintained across failover.

Login

1. Go to the main Sakai portal URL: [http://hostname:port/portal]
2. Login with any user
3. Click on the Resources tool
4. Click on the Home tool
5. Bring the server currently being use down either in a graceful or forceful fashion
6. Click on the Resources tool
7. Click on the Home tool
8. Verify you were not prompted to login again, and both the Resources and Home tool displayed correctly before and after the server was taken down

Worksite Creation

1. Go to the main Sakai portal URL: [http://hostname:port/portal]
2. Login with any user
3. Click on the Worksite Setup tool
4. Click on the Create Master Course Link
5. Select an Institution Title from the drop down
6. Overwrite the text in the Description field to be something you will recognize, e.g., "Cluster test 1234", where 1234 = the test id
7. Press the Continue button
8. Bring the server currently being use down either in a graceful or forceful fashion
9. Press the Back button
10. Verify the text entered in the Description field matches the text entered during step 6