Category: Lecture Notes

We have now arrived at a key concern for the mixed reality user experiences that we have designed: How do we study and evaluate them? What do we define a success criteria? Which research methods make sense?

There are lots of standard methods for evaluating user experiences, and there are specific ones that are most often used in the Mixed Reality research community.

BUT … rather than mindlessly applying the standard methods, we should first think it through for ourselves. And consider how we might appropriate the standard methods for our particular needs. 

The first question we should ask ourselves when evaluating a new human-computer interaction experience is: what are the measures of success in our case?

Of course, we need to be familiar with the toolkit before we can start to pick the right tools to answer this question. So here are some key concepts and case studies to introduce you to the area of research methods for how to evaluate user experiences.

The measure of success

First, we must identify what we want to measure and how we can measure it.

What to measure?

When defining measures of success, it is important to identify which aspects of the user experience or task performance matter most. Depending on the project, this might include: presence and social presence, collaboration performance, creativity, social experience in play, enjoyment, stress, focus or distraction, memory recall, and learning, etc. 

Each of these constructs captures a different dimension of user behavior or experience, and prior work often provides validated ways of measuring them. It is therefore useful to search the literature for established methods and metrics, and bring your ideas to supervision sessions where additional guidance and resources can help refine what you choose to measure.

How to measure it?

To measure these constructs effectively, you typically need to design an experiment in which users complete a specific task while you collect relevant data. This can be done through multiple channels: interaction logging to capture behavior, observations to understand context and patterns, questionnaires to gather structured feedback, and interviews to obtain deeper insights. These methods vary in nature (some provide objective indicators, while others rely on subjective reports), so combining them often gives the most robust understanding of a user experience.

Key concepts and distinctions

Now, there are some key concepts and distinctions that you should know of when designing your user evaluations.

• Objective & subjective measures
  • Objective measures: Observations or data that do not depend on personal feelings or interpretations (e.g., reaction time, error rates).
  • Subjective measures: Data based on personal opinions, perceptions, or self‑reports (e.g., satisfaction ratings, perceived workload).
• Quantitative & qualitative methods
  • Quantitative methods: Research approaches that collect numerical data and use statistical analysis (e.g., surveys with scales, experiments).
  • Qualitative methods: Approaches that gather non‑numerical, descriptive data to understand experiences or meanings (e.g., interviews, observations).
• Dependent & independent variables
  • Independent variable: The factor the researcher manipulates or categorizes to examine its effect.
  • Dependent variable: The outcome that is measured and expected to change as a result of the independent variable.
• Within-subjects & between-subjects designs
  • Within-subjects design: Each participant experiences all experimental conditions.
  • Between-subjects design: Different participants experience different conditions, with each person in only one condition.

Case studies

To ground the above concepts in some specific examples, we will cover two case studies:

• 1) Lystbæk, M.N., Rosenberg, P., Pfeuffer, K., Grønbæk, J.E., Gellersen, H.G. (2022) Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality. ACM Symposium on Eye Tracking Research and Applications (ETRA ’22).
• 2) Grønbæk, J.E.S., Esquivel, J.S., Leiva, G., Velloso, E., Gellersen, H., Pfeuffer, K. Blended Whiteboard: Physicality and Reconfigurability in Remote Mixed Reality Collaboration. In Proceedings of CHI’24.

You can read the details about the research methods in the study/method sections of each paper. But here is a tabular overview of their main differences:

Case study	1) Gaze-Hand Alignment	2) Blended Whiteboard
Study type	Controlled single-user study	Exploratory multi-user study
Purpose	Measuring objective performance and subjective experience	Discovering new collaborative behaviour and subjective experiences
Experimental design	Within-subjects study design (with repetition controlled of tasks)	Within-subjects study design (with open-ended tasks)
Data collection	Collecting questionnaire data on multiple conditions in controlled task	Collecting qualitative subjective feedback on multiple conditions
Analysis	Statistical analysis	Thematic analysis

Example of a between-subjects experimental design

If you’re curious, here’s an example of a between-subjects experimental design, which wasn’t covered in the two main case studies above.

Huang, K. T., Ball, C., Francis, J., Ratan, R., Boumis, J., & Fordham, J. (2019). Augmented versus virtual reality in education: an exploratory study examining science knowledge retention when using augmented reality/virtual reality mobile applications. Cyberpsychology, Behavior, and Social Networking, 22(2), 105-110. https://pubmed.ncbi.nlm.nih.gov/30657334/ 

In this study, researchers used a between-subjects experimental design to compare learning outcomes in AR and VR. Each participant experienced only one medium (either AR or VR), which helped prevent carryover effects—such as participants learning the material once and performing better in the second condition regardless of the medium. This design made it easier to attribute differences in attention, spatial presence, enjoyment, and knowledge retention directly to the technology used rather than to practice or fatigue.

The results showed that VR increased spatial presence, attention, enjoyment, and visual information retention, whereas AR supported better auditory information retention. By separating participants into distinct groups, the study could clearly isolate how each medium uniquely influenced cognitive and psychological processes.

Next up for your group projects

Now, you need to take the above concepts and case studies into account when designing your own user evaluations. Onwards!

The Double Diamond is probably the most well-known innovation framework. It consists of two diamonds, each with a cycle of divergence–convergence.

Our goal for last week and this week is to cover the first diamond – Discover & Define. (The second diamond – Develop & Deliver – is for the remaining weeks of the project.)

Through inspiration materials, we are discovering the challenges and opportunities in the future of Mixed Reality for work. A great place to start is by looking at insights from studies of the workplace and its transformations. Based on that, you can develop a future vision for how Mixed Reality can change it for the better.

Discover: research on future of work

Traditionally, designers go and talk to people affected by specific problems at the workplace. However, in this course, we do it second-hand by looking at inspiration materials and insights from research studies. Another source of inspiration could come from investigating work contexts first-hand, such as our own or our peers’ work situations. Where does MR offer a hammer to the nails in these work environments?

For a condensed catalogue of research on the future of work and its challenges and opportunities, I recommend browsing through Microsoft’s future of work reports. 

These offer scientific insights on issues for work that relate to the inspiration lectures last week. Discover what the data shows about the recent major societal shifts to remote/hybrid work and AI adoption, with these three reports:

• The 2021 report, which focuses on how the shift to remote work (caused by the COVID-19 pandemic) affected different aspects of work, such as team collaboration and creativity. 
• The 2022 report, which focuses on the post-pandemic transition to hybrid work and how it has permanently transformed the workplace experience. 
• The 2025 report, which focuses on how AI is changing work. 

In addition, there are a few recent papers on the future of work in MR that you can explore:

• Productivity in VR (2024)
• Working with Mixed Reality in Public: Effects of Virtual Display Layouts on Productivity, Feeling of Safety, and Social Acceptability (2025)
• Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Public Real-World Settings (2025)
• Augmented reality in-the-wild: Usage patterns and experiences of working with AR laptops in real-world settings (2025)

These insights can complement the inspiration sources shared in last week’s lectures. While exploring the breadth of these challenges, we should diverge in our ideas and cover as much ground as possible – to make sure we “get the right design, before getting the design right” (Buxton, 2007).

Define: envisionment and convergence

After discovering challenges in the workplace and mapping them to opportunities for MR, we can start to define a more specific vision and idea for an MR concept and prototype.

Firstly, envisionment involves selecting among our ideas to converge and define the specific problem and its solution space. The identified problem–solution should be articulated as a future vision for MR to address a problem in the workplace.

Secondly, during our envisioning, there will be unknowns, things that make us curious, or open questions we might have about this prospective future. The goal here is to start to articulate this curiosity. What are the open questions? What do you need to study in order to deliver the best solution to the problem at hand?

Finally, in looking for the right future solutions for a problem, we might also intentionally look for the wrong solutions – to begin to anticipate the future consequences of adopting a novel technology like Mixed Reality at the workplace.

This is where we are offering a wildcard for you to go a completely different path with your project: Any vision for the future of computing also comes with its consequences, and any vision has an anti-vision. In other words, a utopian vision can also have unanticipated dystopian outcomes. In our recently published paper, we argue for adopting a vision-critical perspective and propose to deliberately explore anti-visions of a formulated vision. If you’re interested, you can find the paper here: Grønbæk, J.E., Klokmose, C.N. and Hornbæk, K. How Do Future Visions Shape the Field of Human-Computer Interaction? (CHI 2026)

Once honed in on the vision or anti-vision – and the question(s) you want to address – you are ready to start developing.

Action now: discover & define

This week, you will put what you’ve learned into practice to do the above. You must find the nail (i.e., future of work issue) that goes with the hammer (or “hamMR”) you want to explore. You can start from either side – both are methodically fine from a research perspective. The point is that, when you converge, they should be a good match!

I anticipate that groups will struggle with different aspects of this envisionment part of your project. It is therefore highly recommended that you do the following:

• List all the relevant future of work problems/nails that you find motivating to solve through MR
• Document your experiments with using both paper and video for sketching ideas to address these problems.
• Write a short reflection on the pros and cons of these two different mediums for sketching MR experiences (this will be useful for your report later).
• Create many sketches of ideas to make sure you do not jump on the first idea that comes to mind. Use the entire pile to make your selection for the handin.

Doing the above is the best way to prepare for the the supervision meetings where we will go over your sketches and ideas to help converge on a specific future vision and MR prototype idea.

This week has been project inspiration week. 

I will briefly summarize what you have experienced and give you some perspective on how the content can be operationalised for your project ideation.

We expect you to do group brainstorming with sketches (paper or video) to explore different aspects of how MR can enable new forms of work – for instance, through AI, play/creativity, or multi-device multi-user experiences. More specifically, we expect you to process this week’s content by brainstorming based on each of the three inspiration lectures this week (summarized below).

For now, don’t feel constrained by the technology you have (i.e., Meta Quest headsets) – but rather, focus on the mixed reality user experience. And once we get into group supervision on project idea proposals, we can move from ideation to realisation; i.e., discuss how your visions can be prototyped to start exploring a specific aspect of it.

This week, you were presented by the following challenges and opportunities of MR for the future of work.

XR + AI

Ruofei Du, XR Labs Lead at Google XR, shared his novel experiments on integrating AI into XR, allowing for quick exploration and creation of interactive virtual 3D worlds.

Slides from the talk: link

Recorded lecture: link

Creativity and play at work

Fostering creativity is essential in modern work. And The LEGO Group is at the frontier of innovation when it comes to bringing play and creativity to the workplace.

This week, you experienced how they do at LEGO first hand – through a brick-based tangible activity facilitated by Nick Nielsen, Agile Coach and Change Maker at The LEGO Group.

Nick shared some theory, inspiration, and a play activity. We challenge you to take what he shared and turn it into a novel MR experience for the workplace.

Slides from Nick’s session: link

Multi-device multi-user experiences

In the real-world, it is not enough to make your MR experience work on a single device for a single user. Most use cases involve sharing knowledge and expertise between multiple users across multiple devices.

Sune Wolff, CTO at SynergyXR, was here to share his experiences in building and scaling an XR platform for such use cases with real clients.

Introduction slides: link

Cases: He shared experiences of working with real-world cases with large companies as clients. The following topics were covered (case slides):

• Production line planning (case: Grundfos)
• Store layout optimisation (case: GlaxoSmithKline Shopper Science Lab)
• Remote expert assistance (case: Sanovo)
• Training for hazardous tasks (case: TGS)

While you are not building a prototype for a real client in this course, you can leverage these use cases as motivation for framing the vision for your prototype and identifying MR interaction design problems that you would like to work on.

In design, sketching and prototyping techniques are used at the at the early stages of design to explore a breadth of ideas. It is with such techniques that we are able to envision the future of computing – before building it.

The goal is to arrive at the right solution, before we start to build and refine. 

The problem is that sketching and prototyping for spatial concepts like MR is extremely hard. But the good news is that it’s a skill you can learn. And there are good and general tools to help you, which we can adapt to the purposes of MR design. 

The key distinction that you should know about here is between sketching and prototyping. Some techniques that you will be taught can be used for both purposes, but the main point is that the purpose of sketching is different from that of prototyping. In his book Sketching User Experiences (2007), Buxton articulates this distinction as a continuum (I realise there’s a lot of continua being explained in this course). He expresses the distinction with the following contrasting adjectives at each end:

Source: Bill Buxton (2007). Sketching User Experiences, p. 141.

In this lecture, we will be covering well-established techniques and strategies for sketching and prototyping user experiences in HCI, with an eye toward how they can be adapted to Mixed Reality, specifically. 

Getting the design right and the right design

Before getting the design right, you want to make sure you’ve gotten to the right design. This confusing phrase can be expressed more clearly like this (quoting Buxton, 2007):

• The role of design is to get the right design.
• The role of usability engineering is to get the design right.

Now, how do you get the right design? You find an appropriate tool to sketch out multiple alternatives to arrive at the right solution before getting ahead of yourself and refining a lesser than optimal design.

Sketching (branching, alternatives) vs. Prototyping (iterating, refining) – Source: Bill Buxton. 2007. Sketching User Experiences, p. 388 

Paper Sketching

Now, the classic form of sketching is paper sketches. These are obviously limited when it comes to sketching out MR experiences, which are highly spatial and dynamic in nature. However, they serve as a nice starting point for quickly exploring questions at the early stages of design in a low-effort low-fidelity manner.

Last year, the students of this course also did some sketching exercises to brainstorm new ideas. But with no training, it was clear that this is a skill that should be taught. It’s hard to convey interactive spatial experiences! As a consequence, you will struggle to communicate within your study group (and with your supervisors) about your ideas. 

So I realized we have to teach you how to do it – and after this lecture week, you will be able to express ideas of MR experiences more effectively.

To serve this purpose, we have developed a taxonomy of strategies for MR sketching, which you can explore and use as a visual vocabulary for expressing your ideas at the early stages of MR design. These strategies serve multiple purposes – for externalising your thoughts for yourself, for doing group brainstorming, for communicating with your supervisors, and finally, for creating effective illustration figures in your project reports.

Other Types of Sketches & Prototypes

Now, let’s expand the toolkit a bit beyond pen and paper. 

The taxonomy above focuses on paper sketching, but there are several types of sketches and protoypes that can be used to convey and ideate interaction designs (also listed in the rightmost column of the sketching taxonomy):

Wizard of Oz, Smokes and Mirrors, Bricolage, Word Sketches, Animated Sketches, Video Sketches.

Depending on how they are used, they can placed at different points along the continuum between sketching and prototyping. 

Alternative types of sketching techniques can be useful when you want to express the more dynamic and spatial aspects of user experiences. While there is a wealth of alternative techniques (see the rightmost column here), I will just show you two examples here; the Video Sketch and the Wizard of Oz technique.

Video Sketch example

Buxton (2007) provides an example of a video-based sketch from a student project called Sketch-a-Move. The students sketch out the idea for a novel interactive play experience where a child can determine the path a toy car takes by drawing on its roof.

It is a clear example of a video that has many of the qualities of a sketch (see Fig 1): Quick, Timely, Inexpensive, Disposable.

What appears to be computer-generated movement paths of the car was, in fact, done in with a very inexpensive and simple trick.

How do you think this was done? Well, I’ll tell you in the lecture.. Or you can read Buxton’s book to find out.

The other thing to notice about this video sketch is the visual style, which clearly conveys that it is a sketch and not a product video. Now, contrast Sketch-a-Move with Apple’s promo video for their Knowledge navigator, which fooled the world to think that Apple was working on a product with a personal intelligent assistant. This backfired.

Wizard of Oz example

In the book The Wonderful Wizard of Oz, a curtain is pulled back to reveal that what initially appeared as a mighty Wizard, which Dorothy (the main character) fully believed was real, turns out to be just a small man behind a curtain. But that does not make the experience fake. It felt real to Dorothy – so her experience was real.

In interaction design, we refer to such an experience as a Wizard of Oz prototype. It means we let users interact with a system that appears to work, even though a human is secretly making it work behind the scenes. If we do this well, users can have real experiences with a system before we have fully built it.

Here is an example where we use the Wizard of Oz technique to convey the vision of a shape-changing tabletop display:

• KirigamiTable: Designing for Proxemic Transitions with a Shape-Changing Tabletop (CHI 2020)

The physical table is real and functional. It was built with controllable sliders underneath to dynamically change the shape of a big piece of cardboard (that folds using kirigami principles). However, what appears to be a flexible multitouch-enabled LED display is all fake.

The display is projected (done using dynamic projection mapping with a carefully calibrated multiprojector setup).

And for the input: if you could pull back the curtain, you would see me – with a tablet in the hand – faking the users’ multitouch interactions which were synced to the display, while I verbally instructed the actors to move in synchrony with the motions I enacted on the tablet. (yes, it required a few takes!)

Takeaways about Sketching & Prototyping

What I want you to take away from the above examples is this (paraphrased from Buxton, 2007):

• The fidelity of the experience, not the fidelity of the sketch/prototype/technology, is important for early-stage ideation.
• We can use anything that we want to “fake” such experiences.
• The earlier that we do so, the more valuable it is.

MR Prototyping Tools

Because expressing interaction at the early stages of design is so challenging, we need tools to help us. And there is, in fact, an active research field dedicated to developing better tools for MR prototyping.

I will show you a few examples of research on this from my close colleagues.

• Video prototyping: We can use video prototyping to quickly express AR experiences. (here’s a nice 10-minute presentation by Germán Leiva that elaborates on this concept)

• Prototyping with anchored depth camera recordings: We can also rapidly prototype MR experiences using depth cameras. SpatialProto: Rapid Prototyping of Interactive Mixed Reality

• Programming by Demonstration: With a video prototype and a state inference engine, we can turn a video into an interactive AR app.

These are just a few examples. If you want to read a bit more, here is a nice short and accessible read about the topic:

Michael Nebeling. XR tools and where they are taking us: characterizing the evolving research on augmented, virtual, and mixed reality prototyping and development tools. XRDS (2022)

It also contains a good overview figure:

What did you learn in this lecture?

• Mixed Reality experiences are difficult to sketch and prototype
• You learned that this is an active research area (inventing new tools)
• You learned new skills for how to approach it in your project 
(which you should practice and use)

In this lecture, we will cover the concept of interaction in mixed reality (MR). What is exciting – but also challenging – about this topic is that MR presents an entirely new paradigm of computing interfaces. When digital content is not bound to 2D screens anymore (but rather can exist in the real world around us), the possibilities are endless. And each new possibility brings new challenges. 

It is hard for textbooks to keep up on the topic of MR interaction. Therefore, I have been looking for good literature review papers (recent as well as older ones) that provide a combination of fundamentals and cutting-edge examples of what MR interaction is. 

This list of reviews is for those who are interested (a 2023 review of interaction technique studies, a 2019 review of remote collaborative interaction in MR, and a seminal introduction to 3-D User Interface Design (2001)). 

But as a minimum, I expect you to read this chapter by Jens Grubert (2021) which is intended as an introduction to MR interaction for students. The chapter serves as the foundation for what we will cover here, providing a nice overview of different types of MR interaction techniques. 

MR interaction typology

Here’s a TL;DR of Grubert’s typology, providing an overview of the different types of MR interaction techniques with key examples for each. (I borrowed the typology, but several of the examples are more recent.)

• Tangible interaction: Tangible user interfaces (TUIs) are concerned with using physical objects as medium for interaction with computers. This a common type of interaction in MR, where the visual and tangible experience can be integrated through virtual overlays on the tangible objects. Classic examples include MagicBook (Computers & Graphics 2001) and Urp from MIT’s Tangible Media Group (1999).
• Surface-based interaction: As a special form of tangible interaction, surface-based interaction refers interactions with touch surfaces like mobile tables or entire room surfaces like walls, tables, and furniture. The latter is often enabled in projection-based systems like  RoomAlive (UIST 2014) or ​​WorldKit (CHI 2013).
• Gesture-based  interaction refers to using hand, body, or touch movements as input in MR, allowing users to control systems through sensed physical actions rather than holding dedicated devices. Enabled by cameras and other sensors that track poses and motion, it enables “natural” interactions mid-air, yet a remaining challenge is that prolonged use causes physical arm fatigue. Examples include the hand-based interactions on Meta Quest, or, as a more advanced example, this recent paper on expressive hand gestures: Hand Interfaces (CHI 2022).
• Pen-based  interaction refers to using a stylus or digital pen (often together with a physical surface like a tablet) as a precise input device in MR, enabling tasks such as menu control, note-taking, drawing, modelling, and manipulating 2D interfaces. It offers a more stable and familiar alternative to unsupported mid-air hand gestures or game controllers for fine-grained interaction. Examples include 2D input on tablets – such as in RealitySketch (UIST 2020) or, in 3D mid-air as in various VR sketching apps, like ShapesXR.
• Gaze-based interaction: Gaze-based interaction uses a user’s eye movements as MR input, enabling actions such as selection, navigation, or system adaptation by tracking where someone looks. It can support accessibility (for disabled users) or reduce physical effort by complementing hand input. Techniques vary in speed, accuracy, and suitability depending on the task. As an example, here’s a recent project we did here at AU: Spatial Gaze Markers (CHI 2024)
• Haptic interaction: Such techniques use touch and force feedback to convey physical sensations in MR, stimulating tactile and kinesthetic senses through active or passive devices. It enriches immersion and task performance by providing a sense of physical presence, but faces challenges around portability (as it often requires external devices), visual occlusion (causing tracking issues), and matching virtual feedback to physical objects (requires high precision). A few examples are: Haptic Retargeting (CHI 2016) and HapticBots (UIST 2021). 
• Keyboard and mouse: In MR, text entry is hard! If you don’t sit at your desk, you need ways of providing keyboard input mid-air, such as this example here. They also just released a virtual surface keyboard on Quest. However, if you’re at your personal desk, a physical keyboard could be integrated. Here, keyboard + hand representations matter for the text entry, e.g., Effects of Hand Representations for Typing in Virtual Reality (IEEE VR 2018) —> The Apple Vision Pro also supports – as seen in this video – a nice Augmented Virtuality experience (yeah, now you know what that means!).
• Human-AI interaction: There’s a whole literature review on the intersection of XR and AI. But I link it just to show you the pace at which the development of AI accelerates. The literature review is pre-LLM days (or at least early days), and in recent years, so much new stuff has come out that enables entirely new use cases. Recent examples:  EmBARDiment (IEEEVR 2025), LLMR (CHI 2024), and Thing2Reality (UIST 2025).

One is not enough

Now, the above types of interaction techniques rarely work optimally alone. In other words: One is not enough! They must work together in multiples. This is explored in these emerging concepts of interaction:

• Multimodal interaction combines multiple input and/or output modalities (such as speech, gestures, gaze, touch, or haptics) to leverage their complementary strengths in MR. By coordinating several channels, it aims to improve efficiency, realism, and immersion. Multiple modalities are often better, but the benefits depend on careful task-specific design rather than simply adding more modalities. Examples: The technique Gaze + Pinch Interaction in Virtual Reality (SUI 2017) was developed by Ken and Hans who are from our lab. This was recently implemented into the Apple Vision Pro. It has since been further extended in research, e.g., Reality Proxy (UIST 2025), a technique that relies on object detection, LLMs, and/or digital twin technology. 
• Multi-display / multi-device interaction involves using multiple physical or virtual screens together (ranging from desktops, tablets, and smartphones to HMDs and large display) to expand workspace, support collaboration, or enhance MR experiences. Such techniques are often about enabling content transfer, contextual augmentation, and flexible window management, allowing users to interact seamlessly across devices and reference frames. Examples include FaceDisplay (CHI 2018) and Apple Vision Pro’s EyeSight face display. Other hybrid examples combine multiple computing devices, such as Traversing Dual Realities (CHI 2025), or simultaneous 2D and 3D sketching in MR (e.g., VRSketchIn, SymbiosisSketch). If you’re interested, here’s a large literature review that we conducted on such multi-device MR interaction techniques.
• Multi-user interaction in MR enables multiple people—co-located or remote—to collaborate, communicate, and manipulate shared content in real time. It supports synchronous coordination and joint tasks in a shared 3D space using virtual avatars, to enable social interactions that go beyond what is possible in regular face-to-face communication. A few examples include: Blended Whiteboard (CHI 2024) and A “beyond being there” for VR meetings (2021). 

Techniques vs. Systems

The chapter we have just covered (and expanded) focuses on interaction techniques. Let us zoom out a bit. There is an important distinction in the research field of HCI+MR that you should know of: interaction techniques vs. interactive systems. In short, an interaction technique is a specific implementation that maps input and output modalities to enable users to do actions on objects (e.g., select, manipulate), whereas an interactive system comprises a set of such techniques to achieve a holistic user experience. The important point is that design and evaluation of user experiences can operate at both of these levels. 

Examples to show the contrast: 

• Gaze+Pinch is an interaction technique that is then demonstrated in multiple applications to convey its versatility and rich potential. 
• Blended Whiteboard is an interactive system that comprises multiple interaction techniques to create a unique collaborative experience. 

Before we wrap up, a few 🤯 examples…

Let’s end on a few crazy examples. The goal is to expand your mind on the interaction possibilities of MR (+ show you how fun it is to work in academia):

• MIT Media Lab – Tangible Media Group – Physical Telepresence

• Remixed Reality: Manipulating Space and Time in Augmented Reality
• Ninja Hands: Using Many Hands to Improve Target Selection in VR
• Reality Promises: Virtual-Physical Decoupling Illusions in Mixed Reality via Invisible Mobile Robots

Yes, it is pretty amazing how human capabilities can be augmented when the medium for interaction is able to trick fundamental aspects of human perception!

In this lecture, we will be covering key concepts and taxonomies on the technologies underpinning MR user experiences, covering some specific MR devices, and principal techniques for tracking and displays.

In the past, I’ve found that students can feel this lecture to be overwhelming with just a long string of general concepts and example devices, which has the consequence that nothing really sticks. So this time, I want to flip it around, where we actively look together at a key online resource: https://vr-compare.com/. I will then explain the concepts you need to know in order to compare key devices like Meta Quest 3(S), Apple Vision Pro, Samsung Galaxy XR (Android), Snap Spectacles, and HTC Vive Pro 2 (discontinued in 2025). The AI glasses category (such as the Ray-Ban Meta AI glasses) are not technically an MR dispaly, they are rather a heads-up display (HUD).

My goal with this lecture is that you will learn how to navigate this table which compares the above set of devices – and be able to understand the key differences between state-of-the-art devices and why these differences matter.

In the process of exploring the above multi-dimensional device comparison, we will aim to cover most of the terms below.

Glossary

Devices: 

Each MR device out on the market offers a specific combination of tracking and display technologies, intentionally designed to navigate a set of trade-offs in order to create the best MR experience for their target users. 

In terms of headworn devices, there is a distinction that has become popular in industry:

Headsets vs. Glasses: There is no clear cut, but the distinction is currently used to differentiate heavy MR HMDs (most widely used for high-end home entertainment and gaming) from the lightweight glasses form factors (designed for everyday computing). They map out on the Milgram RV continuum with headsets at the V-end and glasses at the R-end of the spectrum.

Tracking:

In MR, the main modality for tracking, and what we will focus on, is called optical tracking – i.e., techniques that rely on cameras (as opposed to other sensors, like GPS, IMUs, ultrasonic tracking, etc.). Here are some key distinctions to be aware of in optical tracking:

• Marker tracking: Markers can take many forms; retroreflective markers used for IR trackers (like those coming with the Optitrack system), beacons (like the base stations for the HTC Vive Pro 2), barcodes (like QR codes, markers from ARToolkit developed 25+ years ago, or reacTIVision which has often been used for tabletop interfaces like this one). 
• Markerless tracking: In contrast to the marker-based approaches, markerless tracking does not rely on detecting any dedicated markers in the environment, but rather tracks natural features in the environment. There is a range of techniques to enable a markerless approach – such as image texture tracking (e.g., tracking a magazine cover or a coaster), model-based tracking (looking for things where you know the shape in advance, like a Apple’s Face ID), or Structure from Motion (SfM) techniques like SLAM, which is used by headsets to build a 3D scene understanding.
• Model-based tracking: This approach uses a model which it can match to the camera input in order to detect and track the entity that the model represents. Common examples include human face/body/eye tracking and object tracking. 
• ML-based tracking: Modern solutions today are not based on manually made models, but rather on ML techniques that leverage training models on large datasets, such as YOLO.
• Inside-out vs. Outside-in: There are two different hardware approaches to tracking users and objects for MR experiences. Inside-out tracking (explainer video) uses outward facing cameras to track the environment and thereby track the user’s position in relation to it. In contrast, outside-in tracking (explainer video) uses beacons in the environment to track markers on the headset. The general pro/con here is that inside-out is more mobile, whereas outside-in is more precise. Although the distinction is less relevant today (because all recent commercial devices are inside-out), it is still important to know about – especially if you want to do research that requires super precise tracking, like we did in this CHI2016 project here where we had to build our own AR outside-in tracking system using Optitrack cameras and IR markers. 

All of the above tracking approaches make the following modalities available for interaction input: head, torso, hands, eyes, gaze, and more. You will rely on several of them when you get to implement interaction techniques during the course, so it’s good to know about how they thrive and what their limitations are)

Tracking is about how the device gets the right input for interaction. Now let’s switch to the output:

Displays:

Just like there are visual (optical) and non-visual tracking techniques, there is also a breadth out display types and techniques. Beyond the visual type, there are haptic and auditory feedback mechanisms. These are very important in making effective immersive experiences.But for now, we will focus on the visual dimension of MR displays. 

There are two primary things about visual MR displays that you should know: I) we distinguish between two primary types of HMDs with quite different pros/cons, and II) there is a wealth of MR displays beyond HMDs which we will only cover in brief in this course, as you will only be developing for HMDs.

• Two types of HMDs – Optical see-through (OST) vs. Video see-through (VST): OSTs use a visor in front of the user’s eyes and a tiny projector that projects the MR overlay onto the visor. Classic OST examples are Hololens 2, Snap Spectacles, and Magic Leap. In contrast, VSTs have a lens and a screen in front of each of the user’s eyes, and at the outside front of the headset, there are camera pointing out toward the environment, which provide the ‘eyes’ for the user, real-time streaming stereoscopic video to the two screens inside the headset. Classic VST examples are Quest 3, Apple Vision Pro, Samsung Galaxy XR, or any of the modern VR headsets..
• Beyond HMDs: There is a nice display taxonomy by Bimber & Raskar (2006) that visualizes the full range of displays for MR. They range from head-attached, through handheld, to spatial. And there are subcategories in-between. Have a brief read through this paper. A few examples that you should see if you can fit into the taxonomy is SixthSense, the famous interface concept developed at MIT, and RoomAlive by Microsoft Research.

Case studies

We will end these lecture notes with case studies of two MR devices to see how the above techniques have become embedded into modern headsets. We will focus on the Meta Quest 3 and the Apple Vision Pro. They are quite similar on several dimensions, sharing the following main features:

• Hand tracking, face tracking, surface detection, scene understanding (with spatial anchors)
• Recently: Support for pen-based input (Logitech has the MX Ink Stylus for Quest and the Muse for AVP)

However, they also differ in the ways they navigate trade-offs:

Meta Quest 3 (the low-end consumer product)

The latest Quest device is a cheap standalone headset with a fairly open software platform. It is a VST headset with inside-out tracking (see, now you know what that means!). 

Main trade-offs:

• Cons: Display is sufficient resolution (justified by the low price tag), but cannot be used to e.g., read small details or text in the world through the display.
• Pros: The Passthrough Camera API – The video passthrough was recently made accessible for processing by the developer, which makes it ideal for research. And it has enabled some exciting new tracking capabilities, such as marker (QR code) tracking, live (ML-based) object detection+tracking.

Apple Vision Pro (the high-end professional product)

The AVP is another standalone VST with inside-out tracking. But it is a high-end expensive headset aimed for professional use. It’s quite a closed-up platform (it’s Apple! No shock, of course), with only very curated developer access to its tracking capabilities. But it’s a very impressive demo! 

Main trade-offs:

• Pros: Display is very high resolution – best in its class! It even has a display for eyes to support eye contact (although it looks quite goofy). It supports eye tracking for gaze-based interaction and hand tracking with better coverage area, enabling arms to rest in a comfortable position during interaction.
• Cons: It is bulkier/heavier than Quest and more closed-up for developers, e.g., eye tracking and passthrough video is not open enough for research purposes.

As a final remark, in 2025, Google announced a real competitor that sits in between the low- and high-end products is the newly released Samsung Galaxy XR. Check out this quick explainer by MKBHD.

Now, if there’s only one thing you take away…

It should be this:

MR devices have different trade-offs. Know the vocabulary/terms and the differences so that you can choose the right technical solution for the problem at hand.

Mixed Reality (or for short, MR) has transformative potential, especially for modern work and collaboration. We use the term MR to encompass a variety of immersive technologies, differing in terms of device form factor and the user experience they create – ranging from devices that minimally augment reality (AR) to ones that fully immerse the user(s) into a virtual reality (VR). This continuum of MR experiences has a variety of applications, where AR is good for some things and VR for others, and we will cover multiple examples along this continuum.

Mixed Reality for the Future of Work

In the future of work, MR may very well play a crucial role in our productivity and creativity, and collaborative work with colleagues distributed across the world. We can imagine that MR will be available to us as an immersive alternative to our existing devices (desktops, laptops, phones, and smartwatches).

However, the ways in which this technology becomes available more broadly is still unclear. What device form factor will be sufficiently comfortable, capable, and socially acceptable at work? Which hardware and software technologies will be needed to make the breakthrough to adoption? How should users interact with MR content? What is the “killer app” for work? 

These are some of the questions we will grabble with in this course, with the MR technologies available today. Today, MR is available as a range of devices that sit at different points on a continuum of user experiences, which we call the Reality-Virtuality continuum (or simply, the MR continuum).

The Mixed Reality Continuum

The continuum was introduced by Milgram & colleagues (1994) to capture the large variety of user experiences that exist between reality and virtuality.

The range of technologies along the continuum have different pros and cons, which support different use cases.

In this introductory lecture, we will explore a variety of example applications – with focus on the breadth of user experiences (next lecture, we worry about the technical implementations). 

At the “VR end” of the continuum, we have fully virtual worlds. This is the kind of experience that has had most real-world impact today, with immersive entertainment experiences (such as gaming … or even a VR time machine). But it also has work-oriented use cases which has led to recent business cases, such as for professional training, or practicing presentations and high-stakes social interaction). Here, the design goal is to fully immerse the user into the experience so as to forget about their physical reality and adopt the illusion of a new virtual reality – and this illusion can be quite strong! These kinds of experiences are becoming increasingly easy to develop in the AI era, with generative models for world building, such as World Labs, Meta Creator Assistant, and XRBlocks+Gemini (XRBlocks will later be introduced in a guest lecture by the inventor, Ruofei Du from Google). 

At the “AR end” of the MR continuum, the goal is different. It is about situating virtual content in the physical world. This has potential to become the next everyday interface for computing, enabling real-world browsing and navigation, gaming, and learning. But professional applications are starting to find their footing; incl. uses for advanced manufacturing, remote assistance, and immersive conference calling.

Throughout the course, we will cover, in much more detail, some of the use cases for work – along with the underlying technology and interaction paradigms necessary for creating user experiences on the MR continuum.

MR/AR/AV/VR/XR – I’m confused…?

Before we start, let’s get the terminology straight.

Speicher & colleagues (2019) conducted a survey + expert interviews to answer the question: What is Mixed Reality? And the takeaway is that there is no universally agreed upon definition. However, in our course we adhere to the definition visualised in this short explainer video (by the last author of the paper):

• Milgram’s MR continuum extends from Reality (R) to Virtual Reality (VR).
• Virtual Reality (VR) is when you are inside a fully virtual world.
• Augmented Reality (AR) is about adding content on top of the real world.
• Augmented Virtuality (AV) is the inverse of AR, where elements of the real world are added into the virtual world.
• Mixed Reality (MR) involves the merging of real and virtual worlds somewhere along the continuum, which includes AR and AV.
• Extended Reality (XR) is a fancy umbrella term that is very easy to say – and it sounds kind of cool … “XR”. The X is a wildcard for any of the above experiences.