News & Updates
As the 2.6 code freeze approaches, it's a good time to do a gut-check and reflect on where we are and where we're heading from a UX, and in general, a product perspective.
For those of you who don't subscribe to the lists, you might have missed the first UX improvement design project that took place earlier this year. The project's goals were to re-design Sakai to be more web 2.0 friendly with a widget based UI that brings richness, flexibility, and dare I say it... "personality" to the user's experience.
In addition to these lofty changes, the effort was aimed at making it easier (and arguably better) for end users to find and join sites, create a site of their own, and manage their sites.
As is often the case with design, some of these goals were fairly well defined upfront, while others emerged through the creative nature of the process. In the end, our community gained 32 new HTML screens that covered all of the above, and hinted at more.
Alongside this effort was a groundswell of ideation around Sakai 3.0, which promises to make our product the next big thing in academic open-source technology - assuming we can pull it off!
The only thing that stands in our way is us. The challenges we face in working together as an efficient, well coordinated team are not in the least bit trivial. But silver is beginning to line our historic clouds.
Last week, an open meeting took place to discuss how and what we plan to do with the 32 HTML screens produced from the UX improvement project. While the meeting itself had its fair share of confusion, a few brave souls stepped in to commit resources aimed at implementing the screens. Bravo!
Others soon joined in the leap-of-faith, and now, while still understaffed, a small team has come together to tackle this work.
In the meanwhile, progress is being made as the Authoring Summit gets underway, K2 is picking up momentum, and a Sakai roadmap is working hard to get off the ground. At the same time, there's plenty of discussion being had around team building and organizational issues that are key to the herculean effort required for 3.0. All very positive signs!
What's needed now is simply to connect a few UX dots.
So far, the UX screens produced this year have oddly been labeled as 2.6 designs. Given the gravity of the 3.0 vision, as loose as it might be, the knock-on effects (burrowing one of John's lines) of mislabeling the screens has lead some key community contributors, who have an eye on 3.0, to view them as already passé.
This misunderstanding has unfortunately already lead to certain inefficiencies.
To help correct any misunderstandings, I'd like to be clear that these screens were designed with Sakai in mind, not 2.6, 3.0, or any other specific release. The challenges that these designs address are both universal (ex: administration of stuff) and fundamental to Sakai (ex: building, managing, and joining sites).
No matter where we go from here, these issues will remain relatively constant. We'll always need sites and a way to create them. Having a personal dashboard doesn't seem to be going away anytime soon either.
So despite concerns that these screens are passé, imperfect, incomplete, or what have you, the fact remains that they do solve the design issues they intended to solve in a reasonably elegant way. We now have a good solid frame for where the meatier 3.0 stuff can live. In fact, my plan for designing 3.0 is to work within the body of these screens.
Here's a crude but hopefully helpful example to illustrate my point.
So the question is not about whether your institution plans on using 2.6, but rather how efficiently we can produce 3.0? In other words, do we want to be faced with implementing these screens next year, along with all the additional 3.0 screens, or can we stay ahead of the curve by implementing these screens now and avoid worrying about them once the additional 3.0 screens come about?
That's the first dot: These screens lay the UX foundation for 3.0.
The second dot is freeing me up to focus on content authoring and social networking - which requires more resources lending a hand in implementing the UX screens. Currently we need more JS programmers, a committed project manager, and some minor design support.
Assuming that happens, there's still quite a lot of work to be done! From a UX perspective, the heart of 3.0 consists of three large parts:
- A new content authoring paradigm (major mental shift in how we perceive content creation, ownership, and sharing - as well as how existing Sakai functionality is migrated to those contexts).
- A light authoring solution (something needed in the short term - imagine a wysiwyg widget with some ability to pull from tools like Resources.)
- Social networking (a basic framework for ad-hoc user/group interaction that weaves into the content authoring paradigm).
Tangentially, there will likely be a need to re-evaluate the nature of the current Resource tool. Should it continue to be what it is?
The third dot is to understand where OSP fits into 3.0 and Sakai's future in general.
At present, I'm working with the OSP folks to make some 2.6 UX enhancements. While the changes will be relatively minor for this release, they'll hopefully smooth a few sharp corners. More importantly, investing time now will help the OSP folks and me to identify parallels between OSP and the 3.0 vision. From what I can already tell, the parallels are huge!
The last and final dot is all of the other great work being done to change the underpinnings of Sakai to better support the functional parts of the 3.0 vision. Of course I'm referring to K2, but also consider the work that Oxford is doing with hierarchies, the work Cambridge has done with widgets and dashboards, and the various other odds and ends.
So where does this leave us... what's next?
Well for me, the plan is to focus on the following areas over the next few months:
- Keep the ball rolling with implementing the UX screens produced earlier this year. I firmly believe this to be a crucial step in setting the stage for 3.0. Plus, getting beyond this point will results in lots of serendipitous efficiency across the community - I'm confident of that.
- Getting seeped into OSP to help with short-term needs, but more importantly, to plan for bridging long-term 3.0 related goals - which is already a top priority for some of the leaders of that project.
- Working with Jim Eng and other community leaders to foster the content authoring movement which is at the crux of the 3.0 vision. Look for contributions from me on the project planning page as well as the UX planning page (and hopefully, at the Authoring Summit).
- Carving out time to plan for, and do a little sketching around, social networking. I'm playing this one by ear at the moment until some of the other parts I mentioned start seeing some movement.
With a little luck, and your participation, all of this might just come together. No doubt, this is an exciting time for Sakai!
So those are my plans... what are yours?
It recently occurred to me that some of you have questions regarding the general approach that's been taken with the UX Improvement Initiative as well as the current project: Improving the CLE v1.0. I had plans to elaborate on this in Paris, but given the questions I've received, I thought it might be useful if I shed some light on it now.
Initiative vs. project
There are two things happening here: The UX Improvement Initiative and the Projects that fall within it. While "improvement" is fairly broad, the initiative is not a catch-all for any and all UX activity. Instead, the initiative's aim is to improve Sakai in a fairly specific way (more on this further down).
The initiative will use a series of two-month projects, each producing deliverables that have an immediately useful and relevant purpose. In other words, projects will have well defined goals and concrete deliverables intent on solving current design challenges in Sakai.
Take the current project for example. The goals are well defined in the project's profile page and the deliverables will be a set of screens packaged up in a nice bundle I refer to as a "UX Kit" - which basically consists of everything needed for a developer to pick up and implement a design.
Notice that this approach offers two benefits:
The first is that it helps the project focus on tangible results, which keeps things from meandering without a specific outcome. That's not to say that meandering doesn't have its place - just not under this initiative.
The second is that it uncouples design from any dependency on implementation, and vice-versa. This way design can take place on its own schedule and with its own priorities within the context of two-month projects, so long as what's produced can be implemented. This gives UX folks a bit of autonomy in selecting projects and reduces the chance of being bogged down by indecision or delays that aren't directly related to design. In other words, designers can design what they think "should be" without worrying about how and if it will get implemented. If the design has merit, developers can pick it up and implement it at their leisure.
This shifts the emphasis from "code currency" to "quality currency".
Is two months enough time for quality research and design?
While the current project will result in "implementation ready" screens, not all projects under the initiative are required to. Some projects might result in inputs that will be used in other near-term design projects. For example, a project might be research oriented in that it will produce plans, definitions, and tools, but unlike experimental research, the materials produced in this project will be extremely focused.
All deliverables under this initiative should share the characteristics of being relevant to the current state of Sakai, prepared with the intention of being used in a near-term project, and developed to a level of completion that will enable them to be immediately put into practice.
Hopefully by now the focus of the initiative is becoming a bit clearer. The goal is to improve Sakai, not think up something new.
With that in mind, there may likely be a future UX initiative based on thinking beyond the current paradigm. In this future initiative, the projects will consist of more radical and forward thinking R&D that will likely take place over longer intervals with lower rates of screen production than the current initiative. This is marked by the yellow graph below.
The current project
The blue graph represents the current UX Improvement Initiative - with the star icon obviously estimating how far along we are (about ¾ of the way through the first project).
As you see by the angle of the blue graph, the rate of production is quite high. This is explained by the "catch up" effect. Much of today's design work is benefiting from the years of research and discussion that have already taken place. Since we don't need to start from scratch to improve what's already there, we get a nice bump in efficiency.
In terms for what's actually being designed, as you may already know, the current project is mostly concerned with "site setup" and other rudimentary parts of the UX - as opposed to some of the meatier academic aspects of Sakai (that will come later).
That's not to suggest that the work is entirely dry. While rearranging the existing design to flow smoothly, or what I refer to as "plumbing", I periodically toss in a few fresh ideas based on inputs I receive from others, as well as my own thoughts of course. But mostly, the brunt of the effort is plumbing.
Also, since many of the areas touched in the current project are fairly common to other enterprise applications, for example: managing users, setting permissions, changing styles/colors, etc., we can cheat a little by taking advantage of best practices and conventions in place of intensive up-front research. While this doesn't negate the need for usability testing, it does add another bump in efficiency in that we're not dealing with something entirely new and foreign.
Why "site setup"?
Some of you may be wondering why the site setup feature was selected for this first project. Without going too far into the history of how the UX Initiative began, I'll just say that site setup was an area that was discussed early on as an annoyance in the context of new users (in this case, users who have the ability to set up sites) trying to understand Sakai. It was clear from the feedback that this area of Sakai created an unnecessary barrier to those users.
So fixing it made a lot sense from a product perspective.
From a design perspective, I took a close look at how else we might cut into redesigning Sakai and found myself agreeing with "site setup" as an ideal starting point. For one thing, before the site setup process can be redesigned, we'd need to first understand what the resulting site should look like. This line of thinking was fairly attractive in that it removed some of the historic constraints that would sometimes limit a project like this.
Secondly, the site setup process makes an attractive design target since it touches a lot of different areas of Sakai while at the same time acting as a bit of an "add-on" feature – which allows the design effort to go as light or heavy as necessary without committing to more than anyone can handle.
Choosing the first initiative first
If you return to the graphs for a moment, you'll notice that the yellow graph intersects the blue graph where the former picks up and the latter winds down. This yellow graph represents a possible future initiative that will shift the design emphasis from mere improvement to broader and more philosophical questions about Sakai.
Seems like an okay plan, but some of you might be wondering why we don't just start with the yellow graph first and tackle design at a deeper and more rewarding level now in place of the cleanup work that's happening in the first initiative?
For one thing, not all institutions are prepared to accept a quantum leap with respects to changing Sakai. That's not to suggest the result of a deeper design process would force anyone to give up what they currently have, but let's face it, we'd be treading a path less travelled, so all possibilities would be on the table.
Secondly, Sakai needs help now, not two years from now. There's a lot of improvement that can be done in fairly short order with significant gains. Also, circumstantial innovation will undoubtedly be a byproduct of this effort, resulting in a substantially more robust solution than just incremental cleanup work.
Further, any plans for a long-term research project should be considered with caution in this industry since we're often dealing with a rather quickly moving target.
And last but not least, the process of working together as a team with a design lead is new to this community and taking on a theoretically less ambitious effort now will yield huge benefits later on in terms of developing a good team dynamic and building confidence based on our past successes.
Knowing when Sakai has been "improved"
I have a saying that goes: "The more we talk the less we see." It's not a riddle, I mean it quite literally. I firmly believe that regardless of where one is in the product development process, sketching ideas on paper and producing screens to visually support the discussion is invaluable! So the more screens and the faster we can produce them the better off the process.
At some point however screen production will slow. Some of the design work will register with users, and that's what we'll keep. The stuff that doesn't will fall by the wayside. Negative user feedback will be replaced by positive user feedback (or silence). And just as the graphs indicate, the process will reach a lull.
The best way I can describe the max point in the blue graph above is that we'll know we're there in much the same way we came to realize that we have problems in the current product. It'll be fairly obvious.
If that doesn't satisfy you and you're looking for something more concrete, you may want to check out the UX Scorecard I developed. It's basically a set of metrics I've used throughout my career that have proven to give some indication of quality. None of this is six-sigma, but it is a good start if we allow ourselves to put it into practice.
Where we go from here
While all of this is interesting (and hopefully exciting), the long-run success of trying to make a great product requires a shift in world view from being an observer of UX work to being a participant of it. Making lasting and meaningful UX change is by no means a small undertaking. The work is vast, cuts across all areas of the product, and is a feature of our development landscape that requires continuous attention.
I'm thrilled that as a community, Sakai has taken on this agenda, and to keep it moving we need usability folks, designers, developers, and anyone else who wants to see Sakai succeed in this area to get "actively" involved.
So that's it. That's the 50,000 foot view (oops... 15,240 meter view for many of you).
Today the focus is on tackling the low hanging fruit, cleaning up what we have, cranking out screens, and learning from our mistakes along the way.
Once we get to a point where Sakai's UX has improved a bit, we'll shift our focus from "usable" to "useful" and address the bigger strategic questions.
Recently, I had a discussion with Shane Nuessler from The Australian National University where he mentioned it would be nice if I could put a presentation together for the next conference on design analysis, usability testing, and any other methods for understanding how to find and document problems in Sakai's UX using user-centered methods.
I loved the idea but just not sure how my schedule will look in the coming months. So in the meanwhile, I thought I'd take a few minutes and whip together a quick blog post to summarize some of the different methods I've used.
Some of these methods can be used for testing existing products, but I like to think more in terms of testing new designs. Either way, I hope you find them useful.
The trick with surveys is knowing how and what to ask. Without a direct dialog to affirm and discuss the real issues, it's easy to miss the heart of the problem.
But that's not to say they aren't useful. Just try to ask very specific questions like: Do you have dial-up or broadband where you use your system?
Asking general questions like "Do you think the portal is easy to use?" can lead to fairly mixed results.
2. Focus Groups
These are good for general discussions about a product, but they do carry the common pitfalls of group-thought, personality conflicts, shyness, etc.
It also depends on the methodology used. If a group of random people were asked to perform certain tasks in an application, then later came together to discuss their opinions, that could turn out to be a rewarding discussion.
But if the group is lead by a facilitator who walks them through a design and then asks for their opinions, the feedback is fairly tainted since the users weren't given a chance to figure out the UI on their own.
3. One-On-One Interview
When I do these I like to have a few goals in the back of my mind to help keep the discussion moving along – but not much more in terms of preparation. Bringing in a questionnaire usually makes the process too structured to really hear what the user is saying. Instead, I suggest trying to be as apathetic and philosophical with the interviewee as possible. If you can go out and grab a beer with him/her, even better. You really want to try to get them to let their hair down and share issues that go beyond the specifics of the tool, but more into the nature of why they use the tool.
But despite your best efforts, some users just don't know what the problem is. Sometimes they don't realize there are be better ways of doing something than how they're currently doing it. So you might have a lot of users tell you they like something, when in fact the UI or functionality is problematic.
4. Contextual Inquiries (observing people in their own environment)
This is a really nice way to get a glimpse into what a user is doing without relying on them to remember or explain some issue they've had with the software in the past.
If you're only at the design stage with your product, this might not help give you any insight into how your product is being used, but it's still nice to watch a user use other products in their environemtns – particualrly if the user fits your target "persona".
You'd be surprised how shadowing and quietly observing a user can clue you into all sorts of things that spur innovative improvements to software.
5 Heuristic analysis
There's a ton of examples on this all over the web, so I won't re-purpose them here. But I am working on the UX Scorecard which will have a good format for performing these types of reviews.
Basically, the idea is to get a panel of experts together (by experts, I mean designers who have an eye and vocabulary for design heuristics) and have them review the product.
This tends to be a good way to get some very general insight into the some of the common problem areas, but it rarely leads to any deep interaction analysis beyond just screen-level issues. I still think this is a great way to get your feet wet with a fairly nice method for reviewing and documenting a products UI.
6. Usability Testing
Last but not least, good 'ol usability testing. IMO, there is no better way to understand if a design is going to fly or not. There's a lot of different standards in testing – some are even ISO approved. But rather than going the fancy route, I tend to use quick and dirty tests just to get feedback early on while still in a design phase of a project.
When preparing a test, the thing you put in front of users can range from hand paper sketches to a semi-functional html prototype. Whatever is most economic for your organization. I prefer hand sketches myself since you can easily change them in mid-test and they don't take much to prepare.
To get going, I suggest you come up with a set of screens that revolve around a few tasks that you want to test. Don't try to test too much in one sitting as your prototype will get large and unwieldy. Just a few simple screens will do. Once you have your tasks outlined in your mind and have prepared the screens you'll be putting in front of the user, you're pretty much ready to go. Just schedule a few users and have them come in on a day that works well for everyone.
Oh, despite the central limit theorem (having at least 30 observations), Jakob Nielsen seems to think that 5 users is enough:
From my perspective, the statistical accuracy is not as important as simply seeing how users react to a design. From that perspective, 1 user is better than none.
Anyway, if you can get your screens into an electronic format (i.e. sketch them out in Visio, Photoshop, etc.. or scan in hand sketches), you can use a video recording tool like Camtasia to do all the documenting for you. That can be a real joy once you have everything setup properly.
A few things you may want to do to make the testing workout a little better for you:
a) Try to let the user know it's not them but rather the software that's being tested. You may need to reiterate this point a few times... and even still, you'll probably have some user's say things like "Oh, sorry! That was dumb of me!". Just let them know that they found a design flaw and that's exactly what you're looking for.
You may also want to stay away from calling it "user testing" as you're not testing them, but the software – so "usability testing" is possibly a less intimidating way to refer to it.
b) Invoke the verbal protocol (but don't also test for how long it takes for them to complete a task). Basically, keep encouraging the user to think out loud. They'll stop periodically, as it's not natural for them to talk about what they're thinking at every step, but give them a light prod and they should perk back up
c) Try to avoid hinting in anyway! Keep the questions focused on the task or goal. If the user asks how to do something, feel free to play the amateur psychologist and ask "how do you think you should do it?" If they're completely stumped, you've found a design flaw.
Sometimes I setup a system where if the user asks more than once, I ask them if they want to call tech support? If they say "yes", that's when I answer their question. You might have an interim step where you guide them to online help documentation – which makes for a nice way to test two birds with one stone: the product and the help docs.
With usability testing, you could try to focus on the following areas:
- Effectiveness – have they completed the task? This is categoric (yes or no)
- Efficiency – I'm tempted to suggest you avoid this one entirely. But basically, you could look for how long it took them to complete a task, or whether they chose the "preferred" path, etc. This one's a bit contentious though and should certainly be used with caution – especially if the verbal protocol is also used.
- User Satisfaction – at the end of a task and/or the entire test, it's a good time to get some free-form feedback about likes and dislikes.
Anyway, I hope this helps for now. As for how to capture and analyze all this data, that will have to be something I cover in a future post.
I've been steadily chipping away at creating a little place here on confluence, and now that I'm pretty much done, I'm proud to announce that the UX Improvement Initiative has officially been launched!
To summarize things:
- The UX Improvement Initiative is going to make Sakai's user experience awesome!
- We'll make improvements through a series of projects that will target Sakai's UX problem spots. Once a project is formed, we'll keep it open until it doesn't need anymore design. (Where's the skeptical raising one eyebrow smiley when I need it?)
- Projects will get two months of design time, and if they need more, we'll give them another two months at some later time.
- The first project that's been selected is: Improving the CLE 1. The kick-off for this project is planned for Monday, March 17th. I've scheduled a kick-off meeting on the Sakai001 bridge for 9:30 AM MST.
- I encourage anyone who is interested in participating in this project to contact me so I can add you to the team! I'd like to keep the kick-off meeting to 1 hour and will send an agenda to the team this Friday.
- Also, if this project isn't your cup of tea, but you'd still like to get involved in the initiative, let me know and all I add you to the directory!
PS: I've done my best to keep the content as brief as possible. But if you're just not up for the read, feel free to drop me a note with any questions you might have.