This series of 7 short videos gives an introduction to the mixed reality therapy room platform, focusing on providing an overview, demoing the system and process, considering three practical examples and looking at opportunities for further development.

Video 1/7: Overview

Intro

The mixed reality therapy project focuses on supporting the development of psychological therapy by integrating advances that have been made in the areas of experiential technology and simulation.

In certain areas we seem to instinctively understand this idea of taking an experiential approach. If, for example, you were training someone to fly an aircraft you’d recognise that giving them a book on flying wouldn’t be sufficient. It would be necessary to be in a real environment, or one that looked or functioned like it, in order to develop the reflexes necessary so that that learning might transfer to real world scenarios.

Something similar is true of psychological therapy. Taking an experiential approach can serve to help develop skill and stir affect. Of course the core of good therapy is simply being able to be there with the person who needs support, that will never change. But to be able to use experiential technology to scaffold the therapeutic process offers many advantages.

The mixed reality therapy room is a software platform designed to be used by clinicians to support psychotherapy, training and assessment. The platform is neutral to the type of therapeutic methodology used, leaving that to be determined by the practitioner, and serves as an aid to help access the experiential potential that exists in current forms of evidence-based therapy.

The term mixed reality comes from research into display technology, further added to by work on how we perceive media. If we’re taking an experiential approach, and introducing scaffolding to help with that, then there are three main ways to achieve that: physically, virtually and imaginally.

In physical, or in vivo, experiential work the therapy or training takes place in real world settings with other participants where required. The advantages to this approach centre on its ecological validity, however disadvantages include costliness, difficulties in staging certain scenarios and a lack of control of the learning environment.

Another approach is imaginal work. This involves reflecting back on previous experiences, visualising future ones and working through scenarios using mental imagery. Imaginal work is low cost and agile, however it relies on the capacity of participants to be able to adequately imagine the scene, and in some cases extra work may be needed to generalise imaginal learning into real world scenarios.

The third experiential category is the virtual. The virtual exists somewhere between the physical and the imaginal. It lacks the full affordances of a physical environment, however has some limited affordances to help scaffold the experience. Virtual systems, in high or low tech ways, can make use of multisensory cues to support the therapeutic process.

While research often seeks to separate these physical, virtual and imaginal categories (which is useful in order to better understand and compare them), in therapeutic practice it often makes sense to make use of all of the above. Blending the physical, virtual and imaginal in different degrees as needed can serve to create more engaging and effective therapeutic and training environments.

Video 2/7: System

System

The participant accesses the mixed reality therapy room’s virtual environment in 3D by putting on a virtual reality head-mounted display, or in 2D on a computer screen. The platform is built to facilitate both local as well as remote collaboration, as appropriate. Having done so they find themselves in a virtual consultancy room.

This can be just a virtual space, used in its own right, or the virtual space can be layered upon a physical consultancy room to create an even more compelling experience. The consultancy room draws on conventions in therapy room design, in order to feel familiar. It allows for customisation to accommodate individual user differences and variation of therapeutic approach, with different themes that can easily be switched between as needed.

When we look around the consultancy room we can see some of the typical elements we’d expect to find: a couch for the client/patient, a seat for the practitioner, tables and some other furniture. In front of the seating area is a screen. This can be used to show therapeutic content, prepare and review exercises and display data.

Across from the seating area is a doorway that leads into a second adjoining room: the experience room. An important part of the design is the separation of the experience room and the consultancy room. This is a deliberate attempt to create a behavioural distinction between the consultancy room, as that secure base where orientation and reflection can happen, and the experience room, as a place of adventure.

When considering user roles within the mixed reality therapy room, it can be helpful to think of four main types. The first is Participants. These are the clients or patients who are entering the space with a therapeutic or educational goal.

The second role is the Guide. This is the therapeutic practitioner or facilitator. Their role is to guide the participant through the space. In serving this role they straddle an incorporated perspective—being there in the exercise with the participant—and a system level perspective—thinking ahead to what parts of the scenario should be run next.

The third role is the Operator. This role may be filled by the same person as the guide role, or by another person if they would prefer support so that they can focus more fully on working directly with the participant. The operator will keep an eye on what’s happening in the experience from a technical standpoint, loading content and adjusting settings as needed.

Finally, the Architect role is a designer of the space. They are not present during applied work, and serve to maintain and improve the platform’s various processes and resources.

These roles are a helpful way to think about how the space is utilised, but are also very important in terms of the security and build of the platform. Within the experience room a variety of different types of interaction can be facilitated- some more symbolic, and others more photo-realistic and immersive. Because content is being introduced and manipulated, it becomes essential to know who has control over what, and when. For this reason these roles, and safeguards surrounding them, are built into the system from the ground up.

Video 3/7: Process

Process

The distinction between the consultancy room and the experience room allows for a more granular and controlled approach: discussion before experiential work, co-creative work around what might be on the other side of the door, incrementally walking to the door, opening the door, looking through the doorway and returning to the couch in the consultancy room as needed. This can be actual tracked movement, or through the use of a controller while stationary.

Within the experience room itself are options for loading a variety of different objects and immersive scenes, and several ways of interacting with them. This interaction can be virtual, using hand-held motion trackers or the keyboard and mouse of a computer. Or, with a mixed reality installation, virtual assets can be layered upon physical props that are motion tracked.

Users of the platform are visually represented in the space by a simple avatar, which can be further developed if needed. In terms of user roles, the architect curates a database of content that is generally appropriate, as informed by the clinic’s policies and client profiles. The operator, with specific reference to any indications or contraindications clarified during the intake process, loads relevant assets into the space under the guide’s direction. At this point these resources are only visible for the guide, who can then further adjust their features before introducing them to the participant. The participant can then in turn adjust their properties, and constraints can be set to ensure objects vary only along dimensions that are consistent with the therapeutic aims of the scenario.

Interaction takes place through creating objects, and clearing them, locomotion through the space so users can stand in different positions in relation to the elements, the ability to move objects and the ability to scale them. Objects can be coloured in ways that are significant for their meaning, they can be used as a canvas so they can be drawn upon allowing for even more detailed refinement. They can have a mirror texture which can reflect the participant’s avatar and the rest of the scene. Objects can also have a screen texture, allowing for media files to be played upon their surface.

For greater photo-realism, individual 3D items can also be placed in the space. In addition to these objects, immersive scenes can be loaded that transform the experience room into a location. Relevant objects can then be positioned within that scene facilitating more context-sensitive interaction.

Just as a blank spreadsheet doesn’t prescribe how calculations should be carried out, but offers a flexible space to support data processing, in the same way the mixed reality therapy room offers scaffolding for the therapeutic process, but a great deal of range in terms of how that is carried out.

For the sake of demonstration, it is helpful to consider three approximate categories of therapeutic work: work with aversive, addictive and supportive stimuli. In practice a number of stimuli may be utilised and combined in a flexible way in keeping with therapeutic protocols and participant need. Frequently within virtual reality therapy, and other therapies that have a simulated component, the workflow progresses from imaginal work—where topics are explored and mentally visualised—through to virtual work, through to real world practice. There may be multiple loops of this sequence as any challenges are brought back into the therapeutic process, and further explored and worked through.

Video 4/7: Aversive Example

Example 1: Aversive stimuli

Work with aversive stimuli in the platform involves experientially representing and interacting with virtual versions of concrete things, or abstract concepts or feelings, that the participant finds unpleasant and has avoidant or phobic responses to. How treatment will progress will be dependent on the participant’s needs, the style of therapy used and the guide’s direction, but let’s consider some of the options when working with the example of a phobia.

Following the initial intake process and suitability checks, and often some imaginal work to help clarify and complete initial work with the phobic stimuli, the participant and guide can enter the virtual consultancy room. Having defined what aspects of the phobia are most salient, they can spend some time discussing what scenarios might be most helpful to run in the experience room. This makes the process a co-creative one, and is itself a form of imaginal engagement which is typically beneficial in its own right.

For those with more severe phobias, it may then be helpful to use the space between the consultancy room couch and the door to the experience room to allow for incremental steps to be taken, with room to pause, process and bring in psychoeducation as needed. When appropriate, the experience room can then be entered. The experience room may already be populated with relevant stimuli (i.e. a social setting, a stage facing an audience, the top floor of a skyscraper), or it may be entirely empty, depending on which protocols are used. With a driving phobia, for example, it may be the car itself that needs to be represented, or perhaps the presence of other motorists. Or it may be more environmental: a particularly enclosed space, an open one, or one where a lot is going on—leading to sensory overload.

Simple geometric shapes can be used, in a choice of colour or texture, to represent the interior or exterior of vehicles in the space, and their proximity to one another. Or, taking another less literal approach, an underlying feeling such as fear could be reified, or represented, in an object that can then be worked with symbolically. The drawing tools can be used to capture finer details and to provide flexibility and quick visualisation. 3D objects, such as a more photo-realistic virtual car can be brought into the space from the database of assets. Videos of roadways or other relevant content can be loaded, or immersive scenes selected for greater immersion.

In some cases guided engagement with a specific object will be suitable, as in the case of traditional exposure therapy. In other cases, such as work with PTSD, the narrative may be particularly important, and more emphasis placed on the flow between events or the changing spatial location of objects over time. The three dimensional quality of the space gives the participant and guide the opportunity to achieve a great deal relationally. Above and beyond what is represented in the space, how close or far away from it the participant positions themselves, how big or small the assets are scaled and where they are in relation to each other are all central factors in how the scenario is experienced, and can lead to the creation of a strong sense of agency.

Platform safeguards prevent phobic objects from being scaled up or otherwise modified beyond certain agreed-upon limits, and objects are approved by the guide before becoming visible to the participant. There is room to pause and to reflect at any point, to clear content, and to leave the experience room and return to the safe base of the consultancy room whenever needed. After experiential work is completed, the participant and guide return to the consultancy room for a debrief, following which they exit the virtual space.

Video 5/7: Addictive Example

Example 2: Addictive stimuli

Addiction therapy, depending on the therapeutic approach used, may take different levels of focus. If the therapy focuses on dealing more effectively with underlying stressors, which the addiction is being used as a coping mechanism for, work with the platform will more closely resemble the aversion work previously described (and either way the basic workflow will remain the same). On another level of focus, experiential work can be helpful in reenacting moments that the participant has found challenging in terms of resisting addictive urges in day to day life, and highlighting the choice points that sometimes might not be fully noticed or utilised in those moments.

Some addictive stimuli will be a little easier to visually represent, such as a single cigarette or a box of cigarettes for someone who has a smoking addiction. Others might be more complex or situational, like gambling or shopping. The platform provides the option, for complex stimuli, to either break them down into simpler constituent parts, or to represent them symbolically. It may be that the participant already has an iconic evocative way of representing the issue: a scrunched up betting slip or a bill from the credit card company. If not, co-creative work can take place to establish a symbol that either points at relevant addictive stimuli, or to the various thoughts, feelings and behaviours that give rise to the addictive habit.

Taking the example of a smoking addiction, the cigarette or the pack can be depicted in the space using a simple geometric shape, or a 3D model. If context is an important factor, then other assets, or an immersive scene, can be loaded to create a greater sense of place. The participant, with the guide’s support, can practice noticing a choice point at moments in the past when they would have reached for the cigarette. Other more suitable responses can be explored, and also visualised in the space. Choosing a healthier option can be practiced spatially, with the participant moving away from the unwanted stimuli (the cigarette) and towards an alternative choice, creating a physical, spatial and visual experience to help with reinforcement.

Features of objects in the space, such as their scale, colour or proximity to the participant, can also be used as a helpful visualisation tool to indicate the participant’s current affective stance towards that item. The cigarette, for example, may be made to appear much smaller after therapeutic work has taken place, and the positive alternative larger, to represent relative significance.

A greater narrative emphasis might be taken if the guide chooses to use a form of aversion therapy to help create a mindful awareness of the consequences of continuing down a particular addictive path (with a comparison to the positive benefits of following the path of the alternative). The ability to visually depict this in space, and to physically walk that path, can help make what can seem unfamiliar and abstract more concrete and imaginable, and the choice points that lead there more recognisable.

The platform places an emphasis on giving control to the participant and guide, so there is the freedom to make the contents of the experience room as simple or as sophisticated as is needed: from drawing a simple picture on a virtual canvas, through to spatially navigating the consequences of different choices. More elaborate projects will likely be built over time: starting with simple options, then later representing and working through challenges that the participant has experienced between sessions.

Video 6/7: Supportive Example

Example 3: Supportive stimuli

Often a combination of different types of stimuli will be needed to act as counterpoints to each other in experiential therapeutic work. A type of stimuli which can be beneficial in their own right, but also a helpful addition to aversive and addictive work, are supportive stimuli. Supportive stimuli are those that help the participant to access resourceful states: like feeling more relaxed, inspired or reflective.

There are a number of typical scenes that, on average, will be pleasant to be in for most participants: the types of places you’d expect to visualise during meditation – in a forest, by the ocean, in a meadow and so on. These types of scenes are popular for a reason: they do suit a lot of people a lot of the time. However the mixed reality therapy room places a strong emphasis on customisation, so seeks to move beyond off the shelf solutions.

Customisation is important for a number of reasons. Firstly, with a client or patient-centred ethos, many styles of therapy emphasise starting with the participant and working outwards, building a suitable programme of evidence-based support around their needs rather than hoping that they will fit into a one size fits all approach.

Even when there is a reasonable match between participant need and supportive resources, the process of tailoring the environment to the participant’s needs can result in an even more refined fit and a sense of co-creation, with room to modify as needed. It allows diverse needs and accessibility to be taken into account, and contraindications to be accounted for.

As with the workflow for aversive and addictive stimuli, initial work can help clarify what resourceful states are to be targeted, and what types of objects or scenes will be most appropriate. The experience room can then be layered with a relevant scene, or can be entered as an empty space that can be a blank canvas to the creative process of choosing supportive stimuli, carried out by the participant and supported by the guide.

Objects may be simple three dimensional shapes, or more photo-realistic 3D objects. Or they may be an immersive scene, or a combination of scene and objects. This combination provides for a richer sense of immersion and the opportunity for interaction.

The experience room, layered with supportive stimuli, may simply be a nice place to visit to unwind. Or it may be a more active environment, with directed therapeutic practice in mindfully perceiving the space, creating and interacting with objects contained within it.

When used to assist work with aversive and addictive stimuli, supportive stimuli can serve as a useful way to unwind at the end of a session, or as a safe place to transition to if the participant finds the work too challenging. This can be achieved either by entirely switching the scene or by introducing supportive content alongside more challenging elements to help temper their effect. The consultancy room is always available as a neutral space, but supportive stimuli by design have an actively soothing experiential quality. Supportive assets can also be a helpful tool for priming participants before therapeutic work that benefits from a more relaxed, reflective stance.

Video 7/7: Further Development

Further development

The mixed reality therapy room platform is being developed as an open source tool to support the development of evidence-based psychological therapy, training and assessment. The goal at all times is to support transfer of the learning from the virtual environment, though to the participant’s everyday life, through the development of an interesting and effective therapeutic experience. At the platform’s core is personalisation: the utilisation of supportive therapeutic scaffolding that is adaptable in real time to assist in improving the wellbeing of a diverse range of participants.

We’re keen to encourage collaboration between practitioners, researchers and technical and creative specialists. The interdisciplinary nature of a project like this is what makes it challenging, but it’s also its biggest strength.

Practitioners can help by providing feedback and, where and when appropriate, adopting the tool in their practices. Researchers can benefit from using the platform as a base from which to conduct research. Some of the benefits of this include: not having to code environments from scratch; a flexible design space; the ability to easily gather data; the opportunity to build upon existing standardised research; and the satisfaction of knowing that their research, whatever its scale, serves a practical and pro-social purpose in enhancing the efficacy of the platform. Technical and creative specialists can play a crucial role in developing the platform’s code, plugins and asset libraries—creating rich themes that can be layered upon the platform to make it engaging and relevant for participants.

You can help the project by spreading awareness and checking out the specific ways that you can get involved on the website: mixedrealitytherapy.org. There you’ll see access details for the platform, various resources, the latest updates in the project’s development and a development plan listing the planned release and feature milestones that we’re working on.

Thank you very much for your support!