Child pages
  • 2008 Proposed Product Management Process
Skip to end of metadata
Go to start of metadata

New Process Suggestion

  • This is an idea for discussion, please feel free to edit, comment, etc.

Current State

The current requirements process in Sakai, while it results in valuable work getting done, has a number of issues:

  1. It operates almost exclusively from feature requests submitted in Jira, which leaves out many sources of good ideas that could be added to Sakai.
  2. The items that come out of the process do not necessarily have development resources dedicated to working on them. This is frustrating for those who worked on the process and those who voted on the items.
  3. It doesn't result in a coherent roadmap for Sakai that can be communicated to the community (and beyond).

I'm proposing that we either abandon this process or, at the very least, add another process that I will call the "Sakai Product Management" Process.

Goals for Sakai Product Management Process

The major goals I would have for this process are as follows:

  1. To produce a roadmap for the next "Major Release" of Sakai (see the proposed release practices for what this means). Generally, the time line for publishing this roadmap will be 6+ months prior to the major release.
  2. For the community to have a reasonably high degree of confidence that most of the items on this roadmap will be substantially complete by the next major release. This is purposefully squishy. Roadmaps often don't come to fruition, even in commercial enterprises, so I would expect a certain amount of slippage or feature reduction along the way.
  3. To assist in combining like efforts towards a common goal. We need a formal mechanism for identifying common interests in advance of when resources will become available.
  4. To increase visibility about what is being worked on.

Some beneficial effects I would also hope to come out of this would include:

  • The ability to conceive of and plan larger, longer-term projects.
  • Increased cross-institutional tool/code development.
  • A broader set of use-cases informing development, hopefully leading to a better user experience.
  • Longer development timelines, allowing for fuller functional requirements development, more formal usability design and more flexible/abstracted code.

Proposed Process

The general approach I have in mind is:

  1. A Sakai Foundation Product/Project Manager would gather ideas from jira, community interviews and competitive analysis. He would create a list of approximately 15 "medium - large" sized projects targeted for the next major release.
  2. This list would be circulated to the community for comment and would be roughly scoped by technical & functional experts as well as designers in the community. The goal is to give the group a rough estimate of how big each item is relative to the others.
  3. This list is brought to a "product management committee" (or some such) made up of a small number of folks (primarily, but not exclusively) from schools who will be contributing development resources over the next period of time. This group sets the top few priorities and commits to working towards them, ideally with some collaboration between institutions. The foundation kicks in resources where necessary/possible. (Examples of the level I'm thinking about: Common Cartridge support, site/user hierarchy, user interface overhaul of "basic" tools, Improved goal awareness, Improved group awareness...generally things that will take 4+ months to complete)
  4. These priorities (probably 3-5 items from the list) become our published roadmap for Sakai, of course filled in with all the little things that everyone is working on anyways. They would then get more formally designed and scoped and a schedule would get built. Ideally, each of the items becomes a significant part of the "minor" releases envisioned in the release practices proposal. These aren't commitments, but goals. Others around the community are invited to join in the detailed requirements development process around these priority areas.
  5. The committee meets every 6-8 weeks to review progress and discuss what might have changed since the last meeting.

Committee members

The reason for doing this in a small committee instead of simply "on the lists" is (a) to allow a meaningful interactive discussion between folks who have resources and (b) to not have people voting on things that aren't going to get done. This latter things is very frustrating for everyone. I'd rather acknowledge the fact that the work is going to be based on the priorities of the groups with development resources (from design through QA) and put a process in place that collects feedback from everyone but doesn't pretend that those things are going to be what happens, necessarily.

The reason for having a small committee instead of just letting things continue as they are is that I believe (a) we need to start looking further out than it appears to me we have been and (b) we can do a better job of identifying common priorities and lining up resources behind those.

Selection of the committee is problematic and has the risk of being a "star chamber". But, at the end of the day, we should acknowledge the fact that those with resources to commit are going to have the largest say in what happens. Explicitly naming those folks should increase visibility into the process and create a place to provide input. There are a couple of approaches to selecting this group. The one I'm currently favoring is to get an initial group from those schools who are committing "significant" resources for the next major release and have them invite a few others as they see fit.

The mix of skills on the committee is also important. The most important, I think, is some expertise in teaching and learning with technology (functional expertise). Still, there needs to be enough technical skill in some members of the group to understand the general technical approach being proposed for a certain roadmap item. At the risk of naming names, people like John Norman, Clay Fenlason and Stephen Marquard (to pick one person from each of 3 continents) have a good mix of both functional and technical expertise. But we should not be afraid of a few people without technical expertise and a few without teaching/learning expertise - the overall mix is what is important. I also think it is good to include one or two folks from schools without resources - we need that voice if we want to produce a product that is widely used.

There is a risk of this group getting too big for a meaningful meeting. That's a high-class headache, in my view, and if it comes to that we can somehow split the group. We may already have two groups...OSP is already doing something like this. I would imagine they would keep their process focused on portfolios.

In any case, that's enough for now...please add your comments.

14 Comments

  1. I like the aspect of having technical experts evaluate and scope the projects in order to determine how much work is involved with them BEFORE the decision of where to commit resources. This is something I wanted to have in the original requirements process but it did not make it in. I look forward to seeing how this works in the first iteration. (smile)

  2. This looks like a great start to helping to community "get real" about what will get done in each release and allowing for longer term planning and development activities. I like it!

    Along the same line as Aaron's note, should we also include requirements gathering and design estimates?

    1. Yes, that's what I meant. When I say "development" I mean the entire product development life cycle, from requirements development through QA. Instead of "techincal experts" I should have used "specialists" or something.

  3. This sounds very promising. One suggestion: in finding people who can contribute expertise in teaching and learning I'd suggest looking for people who are actively teaching with the Sakai product in totally online courses as well as hybrid courses that supplement face to face teaching. It's also prudent to find people who have taught or are teaching in other learning management systems. While we're not reinventing the wheel, we are doing a fair amount of stuff that has already been done in other LMS/CMSs. Having some familiarity with those products can expedite design issues which sometimes seem, in the current Sakai culture, to be conducted strictly through needs-analysis or ex-nihilo.

    Also, do you mean:

    "But we should be afraid of too few people without technical expertise and too few without much teaching/learning"

    instead of:

    But we should be afraid of a few people without technical expertise and a few without much teaching/learning expertise.

    1. I meant we should not be worried about the particular skills of individual members but rather the overall mix. So we could have a couple who were primarily technical experts and a couple who were primarily functional experts. I've edited the post to correct the error.

      And, yes, the range of uses cases is important. Someone from an institution that has pure distance learning as a major use case would be important, I thnk.

  4. I think you should ask the institutions that Sakai is running at, what they want/need, instead of asking us programmers and admins.

    I have zero facts to base this following conjecture up with, but I feel that the majority of the users of this wiki, and the mailing lists I follow are on the Technical side of sakai, and more involved in running it, compared to the users of the system, who... just use it...

    As such, I think it would be worth while to create a "feature request" protocol that you would allow Institutions to submit as a request, a structured document that answers the things the Sakai developers need to know, as well as giving the space to submit the request in that can be tracked by the Sakai management as a real request... Very often I think this would be spear headed by the technical people who run the hardware/software, and many of the requests would relate to technical issues that are faced by those people... But I think you would also start to see suggestions raised by the actual users (Students/Faculty).

    You would also get a good response out of the Administration/Management of the IT and Academic Depts who deal with Sakai as well as they would see a specific set of rules they would need to follow to actually request a feature.

    Currently where I work, we've received a number of requests for feature improvements, all of which have been filtered through the IT management, and when faced with the fact that consultants cost money (god forbid, I know! we don't have the internal resources to commit to sakai for development projects currently) the request stops at the managers desk.

    For instance, almost all of our professors want an Attendance tracking tool. And unless I missed a random Contrib page somewhere on this wiki, I do not know of one that exists. That request has come in to me, as the Sakai Admin, been bounced up to my manager, and then the CTO, where they ask me to "look for one", where upon I ping the mailing list and search the wiki for anything that looks good. Not finding it, I report back to my bosses, and they say "Maybe we can spend the money on it later..." There are other requests that are floating around our "to-do" list where I work, but until I saw the email pointing me to this thread, I was honestly unsure about how to document those requests.

    If there were a standardized request system, beyond what JIRA provides, I feel that 1. I would use it, and two, it would be an excellent way to attract the comments you are hoping for, as well as a great way to track the overall feeling of the community, from more then the IT staff that runs the systems on each campus.

    How this is executed, I'll leave up to you, but even if its "Answer these 5 things, and email it to ....." its a great start.

    From there I guess a Wiki page could be made, and a Jira entry...

    Just my thoughts.

    Thank you.

  5. I like the idea of Duran very much: make it as easy as possible for the ordinary user to submit a request for a enhanced or new feature of Sakai. This should tie in to the first step in the proposed process ("Sakai Foundation Product/Project Manager would gather ideas from jira, community interviews and competitive analysis"). And Competitive Analysis should also be an important task in this.

    1. I'd add the corollary that it's possibly more important for users to be able to browser and vote/prioritize already submitted requests – otherwise we can easily be overwhelmed by volume. (There are a lot more customers than employees in this shop)

      I think dashboard style tools (most popular, most recent, most activity) might be one possibility. Ideally even some kind of fuzzy heuristic as you're typing (related requests displayed via Ajax shiny-ness maybe).

      I suspect voting/prioritizing/expanding would be even more common than adding as such a library grows in size and maturity.

    2. This could open us up to an avalanche of suggestions, but why not put a "Suggestions?" button on each page that links to a form users can fill out.

  6. Michael et al,

    I think the proposed process would be a giant step in the right direction for Sakai. Having a coordinated, transparent process would facilitate long-term resource allocation and help solidify a more coherent vision for where Sakai as a product is going in the future. Ideally, it should also make it easier for newbies in the community to figure out how to get involved.

    One of the issues I viewed as problemmatic in the previous/existing requirements process was the ability of any user to submit feature requests with varying levels of background, context and detail. Not having the contextual background for a request makes it difficult to articulate exactly what issue the requirement is seeking to address and therefore to problem solve in a collective manner around whether the solution lies in tweaking existing tools/services or in building something new.

    Along these lines, I've been very impressed with the model IU provides of having a faculty committee vet future feature requirements. While the process of having campus gatekeepers who coordinate and communicate Sakai requirements to the larger institution is surely not feasible for all campuses, IU's model and ongoing commitment to sharing these with the community is one we should seek to emulate in soliciting public input from institutions on future Sakai requirements. I note some of this work has already begun and am greatly encouraged by that fact.

  7. I am very supportive of this effort. I have a couple of suggestions/ideas to throw into the mix.

    1. The UX Lead and associate UX community members should be major players in this. As Duran suggests, knowing what the users at different schools need is at least as important as what the developers want.
    2. The UX team could (Fluid is doing this and Daphne and Marc did it for the CM project) spend more time (working with the User support teams) building a stronger network and set of methodologies around gathering user feedback. This feedback could be vetted and discussed as part of a U-Camp or project planning meeting so that the feedback becomes meaningful and projects coalesce.
    3. This kind of leads to my nagging concern about the concept of a committee staffed primarily by the campuses with resources. It makes logical sense to do this, yet it is missing the incentive toward the common good. Are you thinking some campuses would commit resources to the foundation? Right now the schools with committed resources are tending to be driven by local priorities. Much of the time these priorities do meet the needs of a wider audience (though not always implemented in a way that does). I guess this is a question: What does it take to get a seat at the table?
      1. developers developing Sakai tools?,
      2. developers committed to work on whatever the "product team" prioritizes?,
      3. designers designing new tools?,
      4. Good ideas and a comprehensive understanding of user needs?
      5. Or something else?

    Like I said above, I really support this happening and would be fine with answering some of these questions as we move forward with this plan.

    1. A couple of tentative answers....No, I don't see resources being pledged to the foundation at this point in time. I'm not philosophically against it but I think there are other ways to get where we need to go.

      I also wouldn't have the committee consist of only organizations with resources. There would need to be other representation as well and part of the Foundation's job in the process is to keep the broader community concerns front and center.

      You're right to boil this down to "who gets a seat at the table." That's a tough question and there is no perfect. I would like to start with a small group from institutions with resources (from UX to developer to QA) that are committed to working on Sakai (not just locally, but working on aspects of Sakai that will benefit many others) in the next 12 months. I would then ask that group to invite a few others from outside that they would like to work with. We'd review the makeup of the group every year or so.

      1. > I would then ask that group to invite a few others from outside that they would like to work with. We'd review the makeup of the group every year or so.

        This seems very likely to create a "Sakai Club" of insiders. Moodle has this problem and it would be nice if Sakai could somehow avoid it.

        One release a year, review the tech committee once a year – this doesn't sound like a nimble organization to me. It sounds more like the path to stagnation. Ideally, we would capture some of the bottom up aspects of the Apache process that fosters innovation and yet have a group that also looks at the big picture.

  8. The requirements process, as originally conceived, was intended to collect information from a variety of sources and bring it all into a single place where it could be used to prioritize interests of the community. It wasn't really the collection part that broke down (though more work could be done there). Rather, the scope of the information being managed caused the prioritization part of the process to get very unwieldy.

    Since prioritzation was intended to feed into the downstream development process, some of Michael's observations above are correct. However, the requirements process was never intended to create a road map in the sense of "this is what will WILL do". Rather, it was intended to collect what COULD be done, and give some indication of what the larger community was interested in.

    I have always viewed this requirements process as dealing with two kinds of information, actually. The first is suggestions, ideas, proposals, feature requests, and the like. The second is actual requirements, ie., the features and functions of Sakai that MUST be included and MUST work. For historical reasons, this information wasn't captured for early Sakai software development. As a result, we really don't know what most tools or services are supposed to do – not in a documented sense. Furthermore, if we don't know what tools are supposed to do, how can we really test them?

    This formal aspect of requirements was addressed by two efforts led by me, but supported by many people in the community. The first was an attempt to formalize base level requirements for tools. The first document (written by Charles Severance) was Criteria for Provisional Status. This was followed up by Criteria for Supported Status that attempted to document hard tool requirements. A third document, "Criteria for Framework Services", was never written. In addition to these documents, an ad hoc pool of requirements was collected in the Requirements WG Wiki site. The intent was to edit these down into formal requirements for core Sakai tools (and services) in 2008.

    Requirements and documentation have always been afterthoughts for software development, especially in the Sakai project. We have always taken the approach that it's better to see if something will work first, then document it later. I'm speaking in generalities here, some tool projects were handled differently. I certainly understand the pressures that Sakai was up against in the early days of the project – who had time to write design documents? As long as things did get written down, this approach works. Somehow, that wasn't usually the case.

    Sakai is (for better or worse) an open source project. People come and go. Personally, I believe that we have a responsibility as good developers to preserve the thinking and reasons for why we built what we built. If not, then we doom our successors to make the same mistakes that we made in getting to where we are. The good news is that some of our thinking IS preserved, thanks to email archiving of various lists and Confluence. The sad news is that it is horribly organized. Finding things can be a nightmare.

    As we look towards the idea of having a Sakai product manager (which I do think is a GOOD idea), let's try to keep the things I've said here in mind. It's not all about writing code, even good code.