Upgrading Experience, from 2.7 to 2.9
On this confluence page, we will be sharing our experience gained during our upgrade from Sakai 2.7.2 to Sakai 2.9.1 at the University of Murcia - SPAIN (Universidad de Murcia http://www.um.es - https://aulavirtual.um.es). Our goal is to provide some context as to why we have made some changes to the Sakai’s code, that we think that can be useful for other institutions and deployments of Sakai. We will be sharing those features with the community as we move forward in our upgrading plans.
We also want to share our route map so that others facing the same need for upgrade can use them, reduce their time and learn from it. At the same time, we would love to hear from you suggestions about tasks that might also be included in our upgrading process, or your advice for facing those tasks.
This page will be made by the technical staff and it is mainly focused on their concerns around the upgrade process. We also intend to mention all the other staff related to this process and the tasks concerned to them.
After Sakai released its 2.9 version, we set a clear goal for our TEAM, move our deployed version (2.7.2) to the latest one. However, the first step is to write a simple plan, unfortunately for us and everyone in the Sakai community, Sakai upgrades do not arrives by clicking a 'magic button'. The main task concerns technical aspects, we have developed a lot of custom tools and have also changed some pieces of the original Sakai code. But, there are other tasks involved, so we are going to mention all of them at this master plan.
Evaluating Sakai 2.9
What do we really know about the new version? This is our first goal. We weren’t too close to the release of this new version, so we’ll have to review all the new things that comes with it, new properties, new features, new look, and so on.
Look & Feel
This is probably one of the most desired changes from the new version. The neoportal skin modifies some aspects of the general usage at the portal interface and we have to adapt our custom skin to the new portal look and feel. We hope that everybody gets happy with the new changes.
We also have to deeply understand all the new features and properties in order to configure our new environment right. We will need to modify our custom help and tutorial pages. The first thing is to have a look at the release notes confluence page. We are planning to make a custom release note page to notify users about the new functionalities.
We use a wordpress site for our online help, providing our users with short videos and short lessons about every tool that they have available. For the upgrade, We have used that same platform to communicate the new changes. We have made some screenshots with comments to show exactly what's new. Feel free to check them out!!
Major Code Changes
Because we had made custom changes to the original code of Sakai, we could be affected by the new code changes comming in the 2.9 version, not only caused by specific patches in particular tools but also because of the particular way of working that we have with the original code (see Deconstructing Sakai).
How different we are?
Talking about patches we have to collect all the changes made in the original code and classify them. Some changes could be applied to trunk and shared with the community, others maight already be included in Sakai and others might be incompatible with new version. This task give us an idea of the amount of code that we have to rewrite for the new version.
Why are we different?
In the other hand, we are not that different from the rest of institutions, most of the patches that we made solve general problems or add interesting features. Maybe this could be useful for the community. In our first deployment (2010) we were focused on the migration from our homegrown environment, a massive task, and we had no time to spend in contributing our work back to the community, but now we want to do it.
Developing Process. (Mar-Apr, 13)
We created a custom infrastructure to develop with Sakai (see Deconstructing Sakai), so we have to adjust this to the new version. Also we have to rewrite all the necessary patches in order to work in the new version. We hope that our custom tools won’t need major changes to work, as well as contrib tools that we use. All the patches that we considered interesting are going to be contributed, following the necessary steps, jira, cle team call,...
Testing Process. (Apr-May,13)
During the developing process we also do some tests, in order to check that every patch works right, but this is a key task, and we have to do different test with different scenarios. We are going to use a snapshot of our production database to perform functional testing, and this phase are going to spend a significant amount of time. Also we have to test database upgrade scripts.
Checking the mailing lists you will discover some performance issues concerning to the new release (KNL-1011), and that’s the main reason to highly checking performance in our upgrade process. Apparently this issues are already fixed but not included in the 2.9 branch, so far. The next tag for this release could be the definitive tag to install for us.
We want to involve in this process, as much as possible, all our Sakai’s users. Keep them informed about the new features, tools, possibilities, and the schedule will be positive and avoid later rejections about the change into the new version. We like to make a communication plan, something similar Duke's transition communication plan, countdown timers are fun !
The Change. (Before Oct,13)
We don’t have a definitive date to make the final change, so far. It depends on the progress of all the tasks described in this plan. The deadline is 1st Oct. 2013, but we are going to work to make the change as soon as possible.
We are out!!! 31st of July 2013. 2.9.2 is ready at University of Murcia.
In order to understand the reasons for some of the changes that we are going to make in the code you need to get some background of our work. We deployed Sakai from source code at tag 2.7.1, also checking out kernel and indie tools code from the respective tags.
We had to develop some custom tools, and we decided to do it with JSF framework, but Sakai included JSF 1.1, and we wanted to use at least JSF 1.2, so we patched Sakai to get this. Most of our custom tools are developed using JSF 1.2, so we also created a maven archetype to create a new tool.
We have a medium size development team so we need to use some infrastructure to facilitate the daily work, and avoid errors as much as possible. The key tool is Jenkins CI. Developers focus in trunk and Jenkins do the dirty job, compile all the code, merge between branches, deploy to concrete servers, perform unit tests, and so on.
During a year we were working with this infrastructure, modifying Sakai sometimes, and mainly modifying our custom tools, or adding contrib or new custom ones. Beginning of 2012, it’s time for a minor upgrade from 2.7.1 to 2.7.2. We realized that we have worked primarily in our code, but developers and specially Jenkins have to manage a big amount of code that never changes. We missed a less expensive way to change the code, and we thought up one way. The idea was to create a sakai-deploy, similar to kernel-deploy or core-deploy, and obviously an assembly. Once you have this you only need three pom files to deploy Sakai.
But, how to make a change in the code now? You can checkout the maven module you want to change, for example kernel-util. Then make the change and compile, so you have the modified artifact in our local maven repository. And now the trick, we developed a custom maven deploy plugin, based in sakai maven plugin, that look for modules changed locally and replaced them in more complex artifact like war or assembly artifacts, before deploy in tomcat.
During this process we noticed that we could extend this mechanism to all Sakai code. We download binary distribution and create an assembly artifact with the original code of Sakai, and a simple pom to deploy it. With our maven plugin we can detect the artifacts that we changed and replace them by the modified ones.
Role Permissions and Templates
Since the very beginning we have customized almost everything in Sakai to adapt it to our institutional needs. One of the first customizations was the creation of custom site types and custom roles with our own set of permissions. This help us to get a deep knowledge on how Sakai works and brings us full flexibility. It was actually a simple thing to do because we basically studied how Sakai was doing that and then adapted it to our needs. One of the first problems that we faced was to share all that customization with the development team so that each member could deploy (in the easiest way) all those sites, roles and permissions in their local environment.
We came out with the idea of using xml files (realms.xml, sitiosplantilla.xml) + a quarz job (UMUINIT) to populate Sakai with all that information. At first, we only had 6-8 different site types and maybe 6-9 different roles. The file realms.xml was used to set all site.template and group.template with their roles and permissions. After a couple of years, this file was getting bigger and bigger and therefore, close to 'unmanageable'. Last month, realm.xml had 23.000 lines. Here is an example of the xml structure:
<REALM CODE="!user.template.externoUM" DESCRIPTION="Usuario externo de la UMU - puede modificar su contrasenya" TITLE="usuarios externos">
<ROLE CODE=".anon" DESCRIPTION="usuarios fuera aula virtual" />
<ROLE CODE=".auth" DESCRIPTION="usuarios dentro aula virtual">
<PERMISO ID="prefs.add" />
<PERMISO ID="prefs.del" />
<PERMISO ID="prefs.upd" />
<PERMISO ID="user.upd.own.passwd" />
With the migration, we took the opportunity to ' ReThink ' how could we improve this process.
You could check this HTML to see the amount of roles, permissions and templates that we have, and we have to manage.
Fortunatelly we created some pieces of software that reduces the complexity and let us manage this in a simple way. This pieces helps us to reduce errors in the final XML document (realms.xml) and bring up some logical to the permission and rol management.
Now we can add a permission easily by adding in the adecuate permission group, or we can add a new realm, by adding the adecuate roles and permission groups, this save us from many many errors y final result.
Migration tasks are always a nightmare but specially when you forgot about the community and you start to make changes in the source code without any control (You feel like it is your own source code). When the migration/upgrade comes, you are not only migrating a Sakai installation, you must have in mind so many things that could affect you in different ways, and all those problems are only yours...
This migration process has helped us to confirm that our thoughts were right, every change that we need to make in the source code should be contributed to the community. The contributing process will help you in many ways, first you can discover different ways to get your final goal, the way you found to do the tasks could be improved and finally you won't worry about how to fix it in the next migration. This is the way that we have been working during this migration process and the response from the community has been very positive, a lot of our improvements are now included and ready for the next major release of Sakai.
Of course, we have more improvements to contribute, but we know now how to do it in a proper way and we feel more confortable with Sakai, we are not only using but driving Sakai right now.
The conclusions are very positive, and we need to let this feeling flow to the rest of our University and especially to our Sakai users. They need to know that when they are letting us know their suggestions they are improving the global LMS, not only our LMS.
This is the beginning of another journey, we have worked really hard to connect ourselves with the community as much as possible and we want to keep this connection alive. Global thinking is not and option, is a compromise even when you have full access to source code.