In design, sketching and prototyping techniques are used at the at the early stages of design to explore a breadth of ideas. It is with such techniques that we are able to envision the future of computing – before building it.
The goal is to arrive at the right solution, before we start to build and refine.
The problem is that sketching and prototyping for spatial concepts like MR is extremely hard. But the good news is that it’s a skill you can learn. And there are good and general tools to help you, which we can adapt to the purposes of MR design.
The key distinction that you should know about here is between sketching and prototyping. Some techniques that you will be taught can be used for both purposes, but the main point is that the purpose of sketching is different from that of prototyping. In his book Sketching User Experiences (2007), Buxton articulates this distinction as a continuum (I realise there’s a lot of continua being explained in this course). He expresses the distinction with the following contrasting adjectives at each end:
Source: Bill Buxton (2007). Sketching User Experiences, p. 141.
In this lecture, we will be covering well-established techniques and strategies for sketching and prototyping user experiences in HCI, with an eye toward how they can be adapted to Mixed Reality, specifically.
Getting the design right and the right design
Before getting the design right, you want to make sure you’ve gotten to the right design. This confusing phrase can be expressed more clearly like this (quoting Buxton, 2007):
- The role of design is to get the right design.
- The role of usability engineering is to get the design right.
Now, how do you get the right design? You find an appropriate tool to sketch out multiple alternatives to arrive at the right solution before getting ahead of yourself and refining a lesser than optimal design.
Sketching (branching, alternatives) vs. Prototyping (iterating, refining) – Source: Bill Buxton. 2007. Sketching User Experiences, p. 388
Paper Sketching
Now, the classic form of sketching is paper sketches. These are obviously limited when it comes to sketching out MR experiences, which are highly spatial and dynamic in nature. However, they serve as a nice starting point for quickly exploring questions at the early stages of design in a low-effort low-fidelity manner.
Last year, the students of this course also did some sketching exercises to brainstorm new ideas. But with no training, it was clear that this is a skill that should be taught. It’s hard to convey interactive spatial experiences! As a consequence, you will struggle to communicate within your study group (and with your supervisors) about your ideas.
So I realized we have to teach you how to do it – and after this lecture week, you will be able to express ideas of MR experiences more effectively.
To serve this purpose, we have developed a taxonomy of strategies for MR sketching, which you can explore and use as a visual vocabulary for expressing your ideas at the early stages of MR design. These strategies serve multiple purposes – for externalising your thoughts for yourself, for doing group brainstorming, for communicating with your supervisors, and finally, for creating effective illustration figures in your project reports.
Other Types of Sketches & Prototypes
Now, let’s expand the toolkit a bit beyond pen and paper.
The taxonomy above focuses on paper sketching, but there are several types of sketches and protoypes that can be used to convey and ideate interaction designs (also listed in the rightmost column of the sketching taxonomy):
Wizard of Oz, Smokes and Mirrors, Bricolage, Word Sketches, Animated Sketches, Video Sketches.
Depending on how they are used, they can placed at different points along the continuum between sketching and prototyping.
Alternative types of sketching techniques can be useful when you want to express the more dynamic and spatial aspects of user experiences. While there is a wealth of alternative techniques (see the rightmost column here), I will just show you two examples here; the Video Sketch and the Wizard of Oz technique.
Video Sketch example
Buxton (2007) provides an example of a video-based sketch from a student project called Sketch-a-Move. The students sketch out the idea for a novel interactive play experience where a child can determine the path a toy car takes by drawing on its roof.
It is a clear example of a video that has many of the qualities of a sketch (see Fig 1): Quick, Timely, Inexpensive, Disposable.
What appears to be computer-generated movement paths of the car was, in fact, done in with a very inexpensive and simple trick.
How do you think this was done? Well, I’ll tell you in the lecture.. Or you can read Buxton’s book to find out.
The other thing to notice about this video sketch is the visual style, which clearly conveys that it is a sketch and not a product video. Now, contrast Sketch-a-Move with Apple’s promo video for their Knowledge navigator, which fooled the world to think that Apple was working on a product with a personal intelligent assistant. This backfired.
Wizard of Oz example
In the book The Wonderful Wizard of Oz, a curtain is pulled back to reveal that what initially appeared as a mighty Wizard, which Dorothy (the main character) fully believed was real, turns out to be just a small man behind a curtain. But that does not make the experience fake. It felt real to Dorothy – so her experience was real.
In interaction design, we refer to such an experience as a Wizard of Oz prototype. It means we let users interact with a system that appears to work, even though a human is secretly making it work behind the scenes. If we do this well, users can have real experiences with a system before we have fully built it.
Here is an example where we use the Wizard of Oz technique to convey the vision of a shape-changing tabletop display:
The physical table is real and functional. It was built with controllable sliders underneath to dynamically change the shape of a big piece of cardboard (that folds using kirigami principles). However, what appears to be a flexible multitouch-enabled LED display is all fake.
The display is projected (done using dynamic projection mapping with a carefully calibrated multiprojector setup).
And for the input: if you could pull back the curtain, you would see me – with a tablet in the hand – faking the users’ multitouch interactions which were synced to the display, while I verbally instructed the actors to move in synchrony with the motions I enacted on the tablet. (yes, it required a few takes!)
Takeaways about Sketching & Prototyping
What I want you to take away from the above examples is this (paraphrased from Buxton, 2007):
- The fidelity of the experience, not the fidelity of the sketch/prototype/technology, is important for early-stage ideation.
- We can use anything that we want to “fake” such experiences.
- The earlier that we do so, the more valuable it is.
MR Prototyping Tools
Because expressing interaction at the early stages of design is so challenging, we need tools to help us. And there is, in fact, an active research field dedicated to developing better tools for MR prototyping.
I will show you a few examples of research on this from my close colleagues.
- Video prototyping: We can use video prototyping to quickly express AR experiences. (here’s a nice 10-minute presentation by Germán Leiva that elaborates on this concept)
- Prototyping with anchored depth camera recordings: We can also rapidly prototype MR experiences using depth cameras. SpatialProto: Rapid Prototyping of Interactive Mixed Reality
- Programming by Demonstration: With a video prototype and a state inference engine, we can turn a video into an interactive AR app.
These are just a few examples. If you want to read a bit more, here is a nice short and accessible read about the topic:
It also contains a good overview figure:
What did you learn in this lecture?
- Mixed Reality experiences are difficult to sketch and prototype
- You learned that this is an active research area (inventing new tools)
- You learned new skills for how to approach it in your project (which you should practice and use)