Jens Emil 🦥 Grønbæk

Author: Jens Emil

  • WrAIting for CHI

    AI is an amazing research tool. But it is also a tool that has been adopted with such speed in research and writing practices that we’ve had no time really reflect on the questions: What does it actually do to our creativity? Our critical thinking? Or, more specifically, the ability to craft an argument for a CHI paper?

    Beyond having consequences for your thinking, it can also have consequences for your career if you’re not up to date on the relevant guidelines of AI use.

    With that in mind, here are some practical tips for AI use related to CHI submissions, but also a few general points, relayed from other professors, which I find to be good guiding principles. 

    As this topic is constantly in flux, it is important to state the disclaimer that these tips here are gathered in September 2026.

    Tips from professors to students

    Here is a nice post by Eytan Adar about some of the consequences of growing up as a researcher with GenAI:

    AI-Native PhD Students

    In the end of the essay, he’s got some good tips for students:

    • Use GenAI proportionally to how well you know something. The temptation to use it for things you find hard will be high. GenAI will also feed your productive procrastination habits. Fight it.
    • Know which parts of your work are load-bearing. Can you defend a choice, state what would falsify your result, or explain why this analysis and not another one? These are load-bearing decisions and need to be yours.
    • GenAI is sycophantic. It will tell you what you want to hear and won’t tell you when it’s wrong.
    • Ask others to help you with accountability. Your mentors and peers can help keep you honest in your use of GenAI.

    Here’s Niklas Elmqvist’s comprehensive guide on how to use AI in HCI research:

    How to AI: A Guide for HCI Researchers

    I am going to relay this excerpt addressing students:

    Have Some Pride

    You are a Ph.D. student, postdoc, or researcher because you are capable of original thought. Do not trade it away for convenience. It truly is a devil’s bargain. 

    Current cloud-based LLMs are marketed as reasoning at Ph.D. level. But what does it mean for your identity as a researcher if you start acting as though it did? You are not here because you are good at prompting. You are here because you have developed creativity, lateral thinking, judgment, taste, and expertise that took years to build. Even your very human quirks should be valued. Use AI as you would any powerful tool: deliberately, with awareness of what it can and cannot do, and without letting it make decisions that are yours to make.

    Here’s Daniel Buschek’s blog post on writing patterns to avoid – often incurred by overuse of LLMs for writing:

    When LLMs Write Our Papers

    The four writing issues that he highlights are:

    • Marketing Language
    • Performative Related Work Section
    • Incorrect Representation of Cited Work
    • Stretched Result Summaries

    Check out his full essay for examples.

    ACM author policy on AI

    I’ll end on a few practical notes regarding the ACM author policy on responsibility and disclosure of AI use.

    As of the latest ACM author policy, we do not have to disclose use of AI for writing, only when used for doing research and producing other artifacts for the paper. You can read the current policy here: https://www.acm.org/publications/policies/new-acm-policy-on-authorship

    But here’s the important excerpt (as of Sept 2026):

    Re: point 1:

    Here’s a nice tool that helps you to draft a paragraph on AI disclosure for your paper (shared with CHI ACs on an internal channel):

    DAISY

    Re: point 2:

    Despite not having to disclose AI use for writing, remember that we take responsibility for all the writing we submit/publish.

    As a pratical tip, beyond the ones shared by other professors above, you should be particularly worried about fabricated references if you’ve used AI for literature search.

    As an example: if you have used AI to find relevant references, it is very bad practice to cite them without reading those papers… But even worse if you have not even checked whether these references actually exist! Apparently, hallucinated references is a big problem these days, even in fully peer-reviewed and published papers – Nature article: “More than 140,000 fake citations across four research repositories were identified in papers and preprints published in 2025 alone” .

    And there are several news stories about how this has affected researchers’ careers.

    So if you have used AI for finding relevant literature (AI can be a great tool for that!), then make sure:

    1. that the paper exists (and the BibTex reference is valid)
    2. that you read it, of course, so that you can cite it correctly

    🙂

  • Narrative and thematic coherence

    By now, you’ve got the framing right (maybe with support from the Bullet Point Method). 

    But writing a full paper is a different beast. There’s so much rich detail and nuance throughout. And this makes it evermore important to make sure that there’s narrative and thematic coherence throughout the whole paper. 

    The reason it’s so important to get the outline (4-5 bullets) right is because you want the paper to follow this line of argumentation all the way through.

    Moreover, the condensed bullet list (or abstract, if you’ve written it) should represent the most important terminology and keywords that must be properly introduced at the right place in the paper. These keywords tend to be good signals for what you should make sure appears throughout the paper title as well as the section, subsection, and subsubsection headings.

    Said more poetically:

    The abstract is a condensation of the introduction is a condensation of the fullpaper.

    The point is that it eases the reading experience if the narrative structure is coherent throughout: your bullet points provide an outline that comes across at each of these three levels of the paper. 

    You may sometimes want to intentionally break this rule, of course, but in the general case, it is causing friction when the reader is, for instance, promised one structure of the argument in the intro, then presented with a completely different structure in the rest of the paper.

    So, in summary, readers want both narrative and thematic coherence.

    • Narrative coherence is that the reader can follow the same story when they read the abstract, the intro, and the whole paper. This sounds obvious – but when you are drafting your paper, you tend to lose track of this, and you don’t want to submit an incoherent paper.
    • Thematic coherence is that there is a consistent theme in your paper – your terminology stays consistent, and it is clear that the terms you introduce in the beginning help you build the argument throughout, and finally conclude in the end.

    Does your paper have narrative and thematic coherence?

  • Have you got the paper narrative?

    If you struggle to write out the introductory paragraphs of your paper – or if you’ve lost sight of whether your narrative covers the most important points – try:

    The Bullet Point Method ™

    For each paragraph, write out a single line – ideally formulated as a statement. 

    This statement is what the whole paragraph should unpack – motivate, convey the meaning of, and provide evidence for.

    The result of the exercise is a very short 4-5 bullet summary of the paper that you can easily share with your supervisor or other peers. It even helps yourself assess whether you’ve got a complete story or not.

    The good thing about this method is that it is much lower effort to produce this bullet list than writing a full 500-800 word intro.

    So you can treat it as a throwaway artifact. You can produce multiple different ones. 

    And it’s very quick for aligning on the paper story with co-authors in a format where you do not lose sight of the narrative coherence – all while avoiding to overthink the actual flow of the sentences (like you tend to do when writing an abstract).

    You can even do it together with your co-authors in a meeting. It’s fairly quick.

    Have a go!

  • Framing your CHI paper

    Don’t skip the paper writing until the last two weeks before CHI. Because writing is thinking. You should develop the intellectual idea as you do the research – and writing is the best tool to do this. 

    Here’s some of the tips I’ve found most useful to get started writing – focusing on the paper framing, problem-setting, outlining, and introduction:

    1. Identify your key idea and don’t wait to write

    Simon Peyton (inventor of Haskell, formerly at Microsoft) is a brilliant communicator and has this excellent talk:

    How to Write a Great Research Paper

    And his first point is pretty much what I said in the opening. Get started writing. And this process even helps, not just to identify the key idea, but to more clearly articulate it so that it becomes relevant and interesting to the reader.

    What is your key idea? If you only had one sentence…

    2. Create instability/tension for the reader

    This youtube lecture by Lary McEnerny from UChicago is such a gold mine of academic writing advice!

    LEADERSHIP LAB: The Craft of Writing Effectively

    The most important advice in it is: to create value for the reader, you have to really make the problem relevant and interesting to them. And you do this by creating tension for the reader.

    Who is your target reader? What is the value for them? Where is the tension in your paper that creates this value?

    3. Structure the introduction

    When working out the framing for your paper, it is very helpful to start outlining or writing the introduction. There’s kind of a formula to this. A great way to think about it is that you have around four to five paragraphs, and each paragraph serves a distinct purpose.

    Many people have written about this, but here is Niklas’ short essay on it, which sums it up quite nicely: 

    Niklas Emqvist: The Five (or Four) Paragraphs of a Successful Introduction

    What does your four/five paragraphs look like? Sketch them out. Outline them. What is each paragraph trying to do? Is one paragraph trying to make two different points? Well, then split it up.

  • Open PhD call on Human-Centered AI

    Together with my colleague Prof. Clemens Klokmose, I currently have an open call for a PhD position on the topic of Human-AI Interaction for Generative User Interfaces (GenUI).

    Deadline: August 1st, 2026

    If you are interested, have expertise in HCI, and have a strong technical profile, please apply!

  • Mixed Reality User Experiences 2026 – Evaluation

    We have now arrived at a key concern for the mixed reality user experiences that we have designed: How do we study and evaluate them? What do we define a success criteria? Which research methods make sense?

    There are lots of standard methods for evaluating user experiences, and there are specific ones that are most often used in the Mixed Reality research community.

    BUT … rather than mindlessly applying the standard methods, we should first think it through for ourselves. And consider how we might appropriate the standard methods for our particular needs. 

    The first question we should ask ourselves when evaluating a new human-computer interaction experience is: what are the measures of success in our case?

    Of course, we need to be familiar with the toolkit before we can start to pick the right tools to answer this question. So here are some key concepts and case studies to introduce you to the area of research methods for how to evaluate user experiences.

    The measure of success

    First, we must identify what we want to measure and how we can measure it.

    What to measure?

    When defining measures of success, it is important to identify which aspects of the user experience or task performance matter most. Depending on the project, this might include: presence and social presence, collaboration performance, creativity, social experience in play, enjoyment, stress, focus or distraction, memory recall, and learning, etc. 

    Each of these constructs captures a different dimension of user behavior or experience, and prior work often provides validated ways of measuring them. It is therefore useful to search the literature for established methods and metrics, and bring your ideas to supervision sessions where additional guidance and resources can help refine what you choose to measure.

    How to measure it?

    To measure these constructs effectively, you typically need to design an experiment in which users complete a specific task while you collect relevant data. This can be done through multiple channels: interaction logging to capture behavior, observations to understand context and patterns, questionnaires to gather structured feedback, and interviews to obtain deeper insights. These methods vary in nature (some provide objective indicators, while others rely on subjective reports), so combining them often gives the most robust understanding of a user experience.

    Key concepts and distinctions

    Now, there are some key concepts and distinctions that you should know of when designing your user evaluations.

    • Objective & subjective measures
      • Objective measures: Observations or data that do not depend on personal feelings or interpretations (e.g., reaction time, error rates).
      • Subjective measures: Data based on personal opinions, perceptions, or self‑reports (e.g., satisfaction ratings, perceived workload).
    • Quantitative & qualitative methods
      • Quantitative methods: Research approaches that collect numerical data and use statistical analysis (e.g., surveys with scales, experiments).
      • Qualitative methods: Approaches that gather non‑numerical, descriptive data to understand experiences or meanings (e.g., interviews, observations).
    • Dependent & independent variables
      • Independent variable: The factor the researcher manipulates or categorizes to examine its effect.
      • Dependent variable: The outcome that is measured and expected to change as a result of the independent variable.
    • Within-subjects & between-subjects designs
      • Within-subjects design: Each participant experiences all experimental conditions.
      • Between-subjects design: Different participants experience different conditions, with each person in only one condition.

    Case studies

    To ground the above concepts in some specific examples, we will cover two case studies:

    You can read the details about the research methods in the study/method sections of each paper. But here is a tabular overview of their main differences:

    Case study1) Gaze-Hand Alignment
    		2) Blended Whiteboard
    Study typeControlled single-user studyExploratory multi-user study
    PurposeMeasuring objective performance and subjective experience Discovering new collaborative behaviour and subjective experiences
    Experimental designWithin-subjects study design
    (with repetition controlled of tasks)    		Within-subjects study design
    (with open-ended tasks)
    Data collectionCollecting questionnaire data on multiple conditions in controlled taskCollecting qualitative subjective feedback on multiple conditions
    AnalysisStatistical analysisThematic analysis

    Example of a between-subjects experimental design

    If you’re curious, here’s an example of a between-subjects experimental design, which wasn’t covered in the two main case studies above.

    Huang, K. T., Ball, C., Francis, J., Ratan, R., Boumis, J., & Fordham, J. (2019). Augmented versus virtual reality in education: an exploratory study examining science knowledge retention when using augmented reality/virtual reality mobile applications. Cyberpsychology, Behavior, and Social Networking, 22(2), 105-110. https://pubmed.ncbi.nlm.nih.gov/30657334/ 

    In this study, researchers used a between-subjects experimental design to compare learning outcomes in AR and VR. Each participant experienced only one medium (either AR or VR), which helped prevent carryover effects—such as participants learning the material once and performing better in the second condition regardless of the medium. This design made it easier to attribute differences in attention, spatial presence, enjoyment, and knowledge retention directly to the technology used rather than to practice or fatigue.

    The results showed that VR increased spatial presence, attention, enjoyment, and visual information retention, whereas AR supported better auditory information retention. By separating participants into distinct groups, the study could clearly isolate how each medium uniquely influenced cognitive and psychological processes.

    Next up for your group projects

    Now, you need to take the above concepts and case studies into account when designing your own user evaluations. Onwards!

  • Mixed Reality User Experiences 2026 – Envisionment

    The Double Diamond is probably the most well-known innovation framework. It consists of two diamonds, each with a cycle of divergence–convergence.

    Our goal for last week and this week is to cover the first diamond – Discover & Define. (The second diamond – Develop & Deliver – is for the remaining weeks of the project.)

    Through inspiration materials, we are discovering the challenges and opportunities in the future of Mixed Reality for work. A great place to start is by looking at insights from studies of the workplace and its transformations. Based on that, you can develop a future vision for how Mixed Reality can change it for the better.

    Discover: research on future of work

    Traditionally, designers go and talk to people affected by specific problems at the workplace. However, in this course, we do it second-hand by looking at inspiration materials and insights from research studies. Another source of inspiration could come from investigating work contexts first-hand, such as our own or our peers’ work situations. Where does MR offer a hammer to the nails in these work environments?

    For a condensed catalogue of research on the future of work and its challenges and opportunities, I recommend browsing through Microsoft’s future of work reports. 

    These offer scientific insights on issues for work that relate to the inspiration lectures last week. Discover what the data shows about the recent major societal shifts to remote/hybrid work and AI adoption, with these three reports:

    • The 2021 report, which focuses on how the shift to remote work (caused by the COVID-19 pandemic) affected different aspects of work, such as team collaboration and creativity. 
    • The 2022 report, which focuses on the post-pandemic transition to hybrid work and how it has permanently transformed the workplace experience. 
    • The 2025 report, which focuses on how AI is changing work. 

    In addition, there are a few recent papers on the future of work in MR that you can explore:

    These insights can complement the inspiration sources shared in last week’s lectures. While exploring the breadth of these challenges, we should diverge in our ideas and cover as much ground as possible – to make sure we “get the right design, before getting the design right” (Buxton, 2007).

    Define: envisionment and convergence

    After discovering challenges in the workplace and mapping them to opportunities for MR, we can start to define a more specific vision and idea for an MR concept and prototype.

    Firstly, envisionment involves selecting among our ideas to converge and define the specific problem and its solution space. The identified problem–solution should be articulated as a future vision for MR to address a problem in the workplace.

    Secondly, during our envisioning, there will be unknowns, things that make us curious, or open questions we might have about this prospective future. The goal here is to start to articulate this curiosity. What are the open questions? What do you need to study in order to deliver the best solution to the problem at hand?

    Finally, in looking for the right future solutions for a problem, we might also intentionally look for the wrong solutions – to begin to anticipate the future consequences of adopting a novel technology like Mixed Reality at the workplace.

    This is where we are offering a wildcard for you to go a completely different path with your project: Any vision for the future of computing also comes with its consequences, and any vision has an anti-vision. In other words, a utopian vision can also have unanticipated dystopian outcomes. In our recently published paper, we argue for adopting a vision-critical perspective and propose to deliberately explore anti-visions of a formulated vision. If you’re interested, you can find the paper here: Grønbæk, J.E., Klokmose, C.N. and Hornbæk, K. How Do Future Visions Shape the Field of Human-Computer Interaction? (CHI 2026)

    Once honed in on the vision or anti-vision – and the question(s) you want to address – you are ready to start developing.

    Action now: discover & define

    This week, you will put what you’ve learned into practice to do the above. You must find the nail (i.e., future of work issue) that goes with the hammer (or “hamMR”) you want to explore. You can start from either side – both are methodically fine from a research perspective. The point is that, when you converge, they should be a good match!

    I anticipate that groups will struggle with different aspects of this envisionment part of your project. It is therefore highly recommended that you do the following:

    • List all the relevant future of work problems/nails that you find motivating to solve through MR
    • Document your experiments with using both paper and video for sketching ideas to address these problems.
    • Write a short reflection on the pros and cons of these two different mediums for sketching MR experiences (this will be useful for your report later).
    • Create many sketches of ideas to make sure you do not jump on the first idea that comes to mind. Use the entire pile to make your selection for the handin.

    Doing the above is the best way to prepare for the the supervision meetings where we will go over your sketches and ideas to help converge on a specific future vision and MR prototype idea.

  • Mixed Reality User Experiences 2026 – Inspiration

    This week has been project inspiration week. 

    I will briefly summarize what you have experienced and give you some perspective on how the content can be operationalised for your project ideation.

    We expect you to do group brainstorming with sketches (paper or video) to explore different aspects of how MR can enable new forms of work – for instance, through AI, play/creativity, or multi-device multi-user experiences. More specifically, we expect you to process this week’s content by brainstorming based on each of the three inspiration lectures this week (summarized below).

    For now, don’t feel constrained by the technology you have (i.e., Meta Quest headsets) – but rather, focus on the mixed reality user experience. And once we get into group supervision on project idea proposals, we can move from ideation to realisation; i.e., discuss how your visions can be prototyped to start exploring a specific aspect of it.

    This week, you were presented by the following challenges and opportunities of MR for the future of work.

    XR + AI

    Ruofei Du, XR Labs Lead at Google XR, shared his novel experiments on integrating AI into XR, allowing for quick exploration and creation of interactive virtual 3D worlds.

    Slides from the talk: link

    Recorded lecture: link

    Creativity and play at work

    Fostering creativity is essential in modern work. And The LEGO Group is at the frontier of innovation when it comes to bringing play and creativity to the workplace.

    This week, you experienced how they do at LEGO first hand – through a brick-based tangible activity facilitated by Nick Nielsen, Agile Coach and Change Maker at The LEGO Group.

    Nick shared some theory, inspiration, and a play activity. We challenge you to take what he shared and turn it into a novel MR experience for the workplace.

    Slides from Nick’s session: link

    Multi-device multi-user experiences

    In the real-world, it is not enough to make your MR experience work on a single device for a single user. Most use cases involve sharing knowledge and expertise between multiple users across multiple devices.

    Sune Wolff, CTO at SynergyXR, was here to share his experiences in building and scaling an XR platform for such use cases with real clients.

    Introduction slides: link

    Cases: He shared experiences of working with real-world cases with large companies as clients. The following topics were covered (case slides):

    While you are not building a prototype for a real client in this course, you can leverage these use cases as motivation for framing the vision for your prototype and identifying MR interaction design problems that you would like to work on.

  • Mixed Reality User Experiences 2026 – Sketching & Prototyping

    In design, sketching and prototyping techniques are used at the at the early stages of design to explore a breadth of ideas. It is with such techniques that we are able to envision the future of computing – before building it.

    The goal is to arrive at the right solution, before we start to build and refine. 

    The problem is that sketching and prototyping for spatial concepts like MR is extremely hard. But the good news is that it’s a skill you can learn. And there are good and general tools to help you, which we can adapt to the purposes of MR design. 

    The key distinction that you should know about here is between sketching and prototyping. Some techniques that you will be taught can be used for both purposes, but the main point is that the purpose of sketching is different from that of prototyping. In his book Sketching User Experiences (2007), Buxton articulates this distinction as a continuum (I realise there’s a lot of continua being explained in this course). He expresses the distinction with the following contrasting adjectives at each end:

    Source: Bill Buxton (2007). Sketching User Experiences, p. 141.

    In this lecture, we will be covering well-established techniques and strategies for sketching and prototyping user experiences in HCI, with an eye toward how they can be adapted to Mixed Reality, specifically. 

    Getting the design right and the right design

    Before getting the design right, you want to make sure you’ve gotten to the right design. This confusing phrase can be expressed more clearly like this (quoting Buxton, 2007):

    • The role of design is to get the right design.
    • The role of usability engineering is to get the design right.

    Now, how do you get the right design? You find an appropriate tool to sketch out multiple alternatives to arrive at the right solution before getting ahead of yourself and refining a lesser than optimal design.

    Sketching (branching, alternatives) vs. Prototyping (iterating, refining) – Source: Bill Buxton. 2007. Sketching User Experiences, p. 388 

    Paper Sketching

    Now, the classic form of sketching is paper sketches. These are obviously limited when it comes to sketching out MR experiences, which are highly spatial and dynamic in nature. However, they serve as a nice starting point for quickly exploring questions at the early stages of design in a low-effort low-fidelity manner.

    Last year, the students of this course also did some sketching exercises to brainstorm new ideas. But with no training, it was clear that this is a skill that should be taught. It’s hard to convey interactive spatial experiences! As a consequence, you will struggle to communicate within your study group (and with your supervisors) about your ideas. 

    So I realized we have to teach you how to do it – and after this lecture week, you will be able to express ideas of MR experiences more effectively.

    To serve this purpose, we have developed a taxonomy of strategies for MR sketching, which you can explore and use as a visual vocabulary for expressing your ideas at the early stages of MR design. These strategies serve multiple purposes – for externalising your thoughts for yourself, for doing group brainstorming, for communicating with your supervisors, and finally, for creating effective illustration figures in your project reports.

    Other Types of Sketches & Prototypes

    Now, let’s expand the toolkit a bit beyond pen and paper. 

    The taxonomy above focuses on paper sketching, but there are several types of sketches and protoypes that can be used to convey and ideate interaction designs (also listed in the rightmost column of the sketching taxonomy):

    Wizard of Oz, Smokes and Mirrors, Bricolage, Word Sketches, Animated Sketches, Video Sketches.

    Depending on how they are used, they can placed at different points along the continuum between sketching and prototyping. 

    Alternative types of sketching techniques can be useful when you want to express the more dynamic and spatial aspects of user experiences. While there is a wealth of alternative techniques (see the rightmost column here), I will just show you two examples here; the Video Sketch and the Wizard of Oz technique.

    Video Sketch example

    Buxton (2007) provides an example of a video-based sketch from a student project called Sketch-a-Move. The students sketch out the idea for a novel interactive play experience where a child can determine the path a toy car takes by drawing on its roof.

    It is a clear example of a video that has many of the qualities of a sketch (see Fig 1): Quick, Timely, Inexpensive, Disposable.

    What appears to be computer-generated movement paths of the car was, in fact, done in with a very inexpensive and simple trick.

    How do you think this was done? Well, I’ll tell you in the lecture.. Or you can read Buxton’s book to find out.

    The other thing to notice about this video sketch is the visual style, which clearly conveys that it is a sketch and not a product video. Now, contrast Sketch-a-Move with Apple’s promo video for their Knowledge navigator, which fooled the world to think that Apple was working on a product with a personal intelligent assistant. This backfired.

    Wizard of Oz example

    In the book The Wonderful Wizard of Oz, a curtain is pulled back to reveal that what initially appeared as a mighty Wizard, which Dorothy (the main character) fully believed was real, turns out to be just a small man behind a curtain. But that does not make the experience fake. It felt real to Dorothy – so her experience was real.

    In interaction design, we refer to such an experience as a Wizard of Oz prototype. It means we let users interact with a system that appears to work, even though a human is secretly making it work behind the scenes. If we do this well, users can have real experiences with a system before we have fully built it.

    Here is an example where we use the Wizard of Oz technique to convey the vision of a shape-changing tabletop display:

    The physical table is real and functional. It was built with controllable sliders underneath to dynamically change the shape of a big piece of cardboard (that folds using kirigami principles). However, what appears to be a flexible multitouch-enabled LED display is all fake.

    The display is projected (done using dynamic projection mapping with a carefully calibrated multiprojector setup).

    And for the input: if you could pull back the curtain, you would see me – with a tablet in the hand – faking the users’ multitouch interactions which were synced to the display, while I verbally instructed the actors to move in synchrony with the motions I enacted on the tablet. (yes, it required a few takes!)

    Takeaways about Sketching & Prototyping

    What I want you to take away from the above examples is this (paraphrased from Buxton, 2007):

    • The fidelity of the experience, not the fidelity of the sketch/prototype/technology, is important for early-stage ideation.
    • We can use anything that we want to “fake” such experiences.
    • The earlier that we do so, the more valuable it is.

    MR Prototyping Tools

    Because expressing interaction at the early stages of design is so challenging, we need tools to help us. And there is, in fact, an active research field dedicated to developing better tools for MR prototyping.

    I will show you a few examples of research on this from my close colleagues.

    • Programming by Demonstration: With a video prototype and a state inference engine, we can turn a video into an interactive AR app.

    These are just a few examples. If you want to read a bit more, here is a nice short and accessible read about the topic:

    Michael Nebeling. XR tools and where they are taking us: characterizing the evolving research on augmented, virtual, and mixed reality prototyping and development tools. XRDS (2022)

    It also contains a good overview figure:

    What did you learn in this lecture?

    • Mixed Reality experiences are difficult to sketch and prototype
    • You learned that this is an active research area (inventing new tools)
    • You learned new skills for how to approach it in your project 
(which you should practice and use)

  • Mixed Reality User Experiences 2026 – Interaction

    In this lecture, we will cover the concept of interaction in mixed reality (MR). What is exciting – but also challenging – about this topic is that MR presents an entirely new paradigm of computing interfaces. When digital content is not bound to 2D screens anymore (but rather can exist in the real world around us), the possibilities are endless. And each new possibility brings new challenges. 

    It is hard for textbooks to keep up on the topic of MR interaction. Therefore, I have been looking for good literature review papers (recent as well as older ones) that provide a combination of fundamentals and cutting-edge examples of what MR interaction is. 

    This list of reviews is for those who are interested (a 2023 review of interaction technique studies, a 2019 review of remote collaborative interaction in MR, and a seminal introduction to 3-D User Interface Design (2001)). 

    But as a minimum, I expect you to read this chapter by Jens Grubert (2021) which is intended as an introduction to MR interaction for students. The chapter serves as the foundation for what we will cover here, providing a nice overview of different types of MR interaction techniques. 

    MR interaction typology

    Here’s a TL;DR of Grubert’s typology, providing an overview of the different types of MR interaction techniques with key examples for each. (I borrowed the typology, but several of the examples are more recent.)

    • Tangible interaction: Tangible user interfaces (TUIs) are concerned with using physical objects as medium for interaction with computers. This a common type of interaction in MR, where the visual and tangible experience can be integrated through virtual overlays on the tangible objects. Classic examples include MagicBook (Computers & Graphics 2001) and Urp from MIT’s Tangible Media Group (1999).
    • Surface-based interaction: As a special form of tangible interaction, surface-based interaction refers interactions with touch surfaces like mobile tables or entire room surfaces like walls, tables, and furniture. The latter is often enabled in projection-based systems like  RoomAlive (UIST 2014) or ​​WorldKit (CHI 2013).
    • Gesture-based  interaction refers to using hand, body, or touch movements as input in MR, allowing users to control systems through sensed physical actions rather than holding dedicated devices. Enabled by cameras and other sensors that track poses and motion, it enables “natural” interactions mid-air, yet a remaining challenge is that prolonged use causes physical arm fatigue. Examples include the hand-based interactions on Meta Quest, or, as a more advanced example, this recent paper on expressive hand gestures: Hand Interfaces (CHI 2022).
    • Pen-based  interaction refers to using a stylus or digital pen (often together with a physical surface like a tablet) as a precise input device in MR, enabling tasks such as menu control, note-taking, drawing, modelling, and manipulating 2D interfaces. It offers a more stable and familiar alternative to unsupported mid-air hand gestures or game controllers for fine-grained interaction. Examples include 2D input on tablets – such as in RealitySketch (UIST 2020) or, in 3D mid-air as in various VR sketching apps, like ShapesXR.
    • Gaze-based interaction: Gaze-based interaction uses a user’s eye movements as MR input, enabling actions such as selection, navigation, or system adaptation by tracking where someone looks. It can support accessibility (for disabled users) or reduce physical effort by complementing hand input. Techniques vary in speed, accuracy, and suitability depending on the task. As an example, here’s a recent project we did here at AU: Spatial Gaze Markers (CHI 2024)
    • Haptic interaction: Such techniques use touch and force feedback to convey physical sensations in MR, stimulating tactile and kinesthetic senses through active or passive devices. It enriches immersion and task performance by providing a sense of physical presence, but faces challenges around portability (as it often requires external devices), visual occlusion (causing tracking issues), and matching virtual feedback to physical objects (requires high precision). A few examples are: Haptic Retargeting (CHI 2016) and HapticBots (UIST 2021). 
    • Keyboard and mouse: In MR, text entry is hard! If you don’t sit at your desk, you need ways of providing keyboard input mid-air, such as this example here. They also just released a virtual surface keyboard on Quest. However, if you’re at your personal desk, a physical keyboard could be integrated. Here, keyboard + hand representations matter for the text entry, e.g., Effects of Hand Representations for Typing in Virtual Reality (IEEE VR 2018) —> The Apple Vision Pro also supports – as seen in this video – a nice Augmented Virtuality experience (yeah, now you know what that means!).
    • Human-AI interaction: There’s a whole literature review on the intersection of XR and AI. But I link it just to show you the pace at which the development of AI accelerates. The literature review is pre-LLM days (or at least early days), and in recent years, so much new stuff has come out that enables entirely new use cases. Recent examples:  EmBARDiment (IEEEVR 2025), LLMR (CHI 2024), and Thing2Reality (UIST 2025).

    One is not enough

    Now, the above types of interaction techniques rarely work optimally alone. In other words: One is not enough! They must work together in multiples. This is explored in these emerging concepts of interaction:

    • Multimodal interaction combines multiple input and/or output modalities (such as speech, gestures, gaze, touch, or haptics) to leverage their complementary strengths in MR. By coordinating several channels, it aims to improve efficiency, realism, and immersion. Multiple modalities are often better, but the benefits depend on careful task-specific design rather than simply adding more modalities. Examples: The technique Gaze + Pinch Interaction in Virtual Reality (SUI 2017) was developed by Ken and Hans who are from our lab. This was recently implemented into the Apple Vision Pro. It has since been further extended in research, e.g., Reality Proxy (UIST 2025), a technique that relies on object detection, LLMs, and/or digital twin technology. 
    • Multi-display / multi-device interaction involves using multiple physical or virtual screens together (ranging from desktops, tablets, and smartphones to HMDs and large display) to expand workspace, support collaboration, or enhance MR experiences. Such techniques are often about enabling content transfer, contextual augmentation, and flexible window management, allowing users to interact seamlessly across devices and reference frames. Examples include FaceDisplay (CHI 2018) and Apple Vision Pro’s EyeSight face display. Other hybrid examples combine multiple computing devices, such as Traversing Dual Realities (CHI 2025), or simultaneous 2D and 3D sketching in MR (e.g., VRSketchIn, SymbiosisSketch). If you’re interested, here’s a large literature review that we conducted on such multi-device MR interaction techniques.
    • Multi-user interaction in MR enables multiple people—co-located or remote—to collaborate, communicate, and manipulate shared content in real time. It supports synchronous coordination and joint tasks in a shared 3D space using virtual avatars, to enable social interactions that go beyond what is possible in regular face-to-face communication. A few examples include: Blended Whiteboard (CHI 2024) and A “beyond being there” for VR meetings (2021). 

    Techniques vs. Systems

    The chapter we have just covered (and expanded) focuses on interaction techniques. Let us zoom out a bit. There is an important distinction in the research field of HCI+MR that you should know of: interaction techniques vs. interactive systems. In short, an interaction technique is a specific implementation that maps input and output modalities to enable users to do actions on objects (e.g., select, manipulate), whereas an interactive system comprises a set of such techniques to achieve a holistic user experience. The important point is that design and evaluation of user experiences can operate at both of these levels. 

    Examples to show the contrast: 

    • Gaze+Pinch is an interaction technique that is then demonstrated in multiple applications to convey its versatility and rich potential. 
    • Blended Whiteboard is an interactive system that comprises multiple interaction techniques to create a unique collaborative experience. 

    Before we wrap up, a few 🤯 examples…

    Let’s end on a few crazy examples. The goal is to expand your mind on the interaction possibilities of MR (+ show you how fun it is to work in academia):

    Yes, it is pretty amazing how human capabilities can be augmented when the medium for interaction is able to trick fundamental aspects of human perception!