Managing Local Modifications

Approaches

There are 2 main approaches in wide-use within the Sakai community to manage local modifications to Sakai code, and incorporating updates to externally produced components

• Vendor Branches
• Overlays

Vendor Branch Approach

In the vendor branch approach, an area of a local SCM repository (typically SVN within the Sakai Community) is is generally split into at least 2 areas, a "mainline" representing the local version of the code, and a "vendor" tree dedicated to mirroring an external project. This area is updated periodically through regular "vendor drops" – exports from the upstream project, which are used to track changes. These changes can then be merged into the "mainline" branch to update it with any fixes, etc. in recent updates.

Strengths:

• SCM-assisted merging and diffing
• (more) easily manage large, or complicated local modifications

Weaknesses:

• Complete copy of external projects maintained in local repository
• Changes tend to get made locally, require extra effort to push upstream
• less visibility over local modifications to a source tree

Overlay Approach

In the overlay approach, a release from an upstream project is typically downloaded/exported, then combined with a local collection of files which have been changed. Local versions are used to replace upstream versions, building a file incorporating local modifications.

Strengths:

• simple visibility on locally changed files

Weaknesses:

• incorporating vendor changes to locally modified files can be difficult

Individual Experiences

Various individuals have contributed their experiences with managing both Sakai and local modifications.

Sean Keesler

I am wrestling with the "best" way to make local modifications to
Sakai for our local install using a local svn repository. As I
understand it, there are a few ways that folks can use to make local
mods.

1. Export a tagged release of the Sakai source and maintain an
"overlay" in a local Subversion repository. This is a file structure
that contains the changed files for the local install.
pros:

• great if there were only changes to existing files (config
  files) or additions to the tagged release
  (a new skin, a locally maintained tool or local
  patches for a tool not yet included in the community bundle).
  cons:
• an overlay can't remove files from the source (can it?)
  (maybe we don't want to compile tool x for our own install).
• requires a 2 step process for checking out the local
  install code (checkout the tag and then apply the overlay)

2. Use svn externals in a local repository and maintain an overlay.
pros:

• code isn't duplicated in our own repository and we can
  select only the Sakai projects that we want in our own install
• we can still maintain local changes to tools and new
  stuff (skins, local tools, local patches) in the overlay
  cons:
• it is still a two step process

3. Export a tagged release of the Sakai source and import it into the
local svn repository as a "vendor drop" (http://svnbook.red-bean.com/
nightly/en/svn.advanced.vendorbr.html). I'm new to this one, but I
think it sets up a structure like the following in your svn repository:

pros:

• lots of local control to include or not include pieces
  and parts in the locally tagged code
• no overlay needed, 1 step checkout
  cons:
• local duplication of the community code (as opposed to
  externals)

Are these pretty much the options available? Can anyone add pros/
cons to this list? Success/failures?

Clay Fenlason

I think the one big thing you're leaving out of your picture is the
application of bug fixes. In production you'll want to either:

a) surgically apply particular patchesi disa
b) synch up with the latest state of the maintenance branch
c) redeploy the latest maintenance release underneath.

.. and perhaps a combination of these, when you come across critical
fixes for which you can't afford to wait.

In short, it's not the application of customizations that's the sticky
part, it's maintaining those customizations in parallel with patching
the codebase underneath. I think this is where the vendor branch
starts to look a little less appealing, what with the extra merging
and potential for conflicts (though that danger may be minimal). The
overlay approach seems to make it a little easier to handle with a
build script (so that "extra steps" are negligible).

I'd like to hear more about this, since we don't have any manual
steps, and perhaps that means we've overlooked something. Or perhaps
some of this depends on the kinds of changes you're talking about, and
ours are relatively straightforward.

We simply check out the codebase from source.sakaiproject.org, then do
additional --force exports from a local subversion repo, overwriting
and adding where necessary. All this is handled by a single build
script (also in version control), shuffling the two repos together as
appropriate, then building.

> Also, we found that with the overlay, we found as many merge conflicts as
> with the vendor drop.

I can certainly see why that would be, and my initial false sense of
things I think comes from that fact that we overwrite rather than
merge. Or, better put, our vendor drops happen module-by-module,
rather than for the entire source tree. Our skin person handles
reference/library. A sys admin handles all the tool registration and
config files, along with the build script itself. A developer handles
an extra contrib tool. Each has its own branch, and produces its own
tags, which the build script pulls together. In some sense (although
we haven't yet quite gone about it this way), what we're really doing
is the externals approach, along with the sort of personalization
tweaks everyone needs to do.

> The advantage of the vendor drop is that you are letting svn handle
> branches, original revisions, merges, etc just like it is designed to do.
> Svn has a script that is designed to check in a vendor drop, then import a
> future version of that vendor drop. For moved files, you have to tell it
> where they moved to. But you would have to do that in an overlay as well.

Right. Again, I now think our approach might be better termed a very
selective vendor drop, piecemeal by module, for the handful of modules
we need to customize. We try to minimize the things we change.

> In either case, however, you should make a very strict "extend where
> possible" rule to avoid merge conflicts.

Absolutely. Our first provider simply replaced the
SampleUserDirectoryProvider, under the same name. Our first skin
overwrote the "default." We've grown a little wiser since then.

John Ellis

I think the problem for us is that we have needed to make a few more source
code changes. Also, we don't always start with a "final" version. Often,
we have had to start with some rc and keep up to date with the various tags
as we are developing. This means that if we change a .java file, then it is
changed with the next tag, if we don't merge the changes together, we will
loose (because we are overlaying a previous version) the fixes that have
been made between the tags. When the developer that needs to change a file
checks that file into the overlay, they must first check in a pristine
version of the file, otherwise, the merge operation above would be near
impossible.

For us, the overlay method has another drawback. It is very tricky for
developers to setup development environments. They need the original
community source, they need a directory they can build from and the need the
workin directory of the overlay. With vendor drops, they just check out the
branch of community source that matches the project they are working on. If
they need to make a change, they just check that change in.

We try to minimize the things we change as well, but the way the tool
registration is designed means we effectively need to change something in
almost every project.

When working with vendor drops, if you haven't changed things locally, svn
merge will notice that and not cause you any manual effort, so there isn't
additional work here than with the selective process you've defined below.
Basically, we have a vendor drop area that is an exact mirror of the latest
tag. From there, we branch off for projects (rsmart cle 2.1.2, 2.2,
customer XYZ, etc). When we need to take another drop of sakai code, we
check that into the pristine mirror, then run an svn merge to our different
projects. In those projects, anything we have customized comes up with a
merge conflict (just as you would find with the overlay technique). The
difference is that svn knows the community tag start point and end point and
the branch point, so it is able to make decisions for us to help with
merging.