Over millions of years, evolution has shaped our human bodies and brains for sensing, acting and thinking in the physical or actual world. To probe, handle and assess food – a necessity for survival – hands and lips played a key role in tactile sensing. The highest density of receptors can be found in our lips and fingertips, despite their small physical size. Consequently, the proportion of the brain dedicated to tactile sensing is relatively large. As a thought experiment, let’s assume that the relevance of our individual senses for acting in the actual world corresponded to the proportions of the brain dedicated to their processing. What if we could visualize these proportions in an intuitively accessible way? Meet Penfield’s homunculus, a deformed human shape in which the size of body parts is changed to match those proportions (Box 1 and Figure 1). It seems evolution has prepared us well for the physical world – but how so for the metaverse?

More and more people will likely commute into virtual worlds on a regular basis with the increasing availability of virtual reality devices in consumer markets. However, for the time being, virtual environments only deliver sensory stimulation to a very small subset of human sensory capabilities. The virtual experiences provided today are predominantly visual. If you are lucky, they may be spiced up with some acoustic events. No olfactory, no gustatory and hardly any tactile or kinesthetic stimuli are provided in a systematic manner. If we were to design a homunculus that depicts the adaptation required for the metaverse, it would be quite different from Penfield’s homunculus. In fact, our “Homunculus Metaversensis” for virtual reality could look like the one depicted in Figure 2. Let´s take a closer look at the current “sensory” state of the metaverse and the technologies that stimulate our senses in virtual environments.

While the direct perception of tactile stimuli and our own body posture is essential for our acting in the actual world, our information society is driven by visual and auditory perception. It is thus no wonder that media technology has focused on delivering stimuli to our visual and auditory systems from past to present as part of communication media. Virtual and augmented reality technologies such as head-mounted displays like the Varjo XR-3 and the Valve Index, which often combine visual and auditory displays, are ready to continue this success story into the future and are the stepping stones of the metaverse. Advances in computing technology, in particular graphical processing units, together with advanced display technologies embedded in current state-of-the-art head-mounted displays, allow us to experience photorealistic digital worlds at resolutions in which individual pixels are no longer distinguishable by human eyes.

The fundamental difference between virtual and augmented reality and other types of media is the immersion of the human into the digitally generated world, both in terms of both sensory functions and motor functions. Different tracking technologies allow us to translate the movements of our heads and hands into movements in the digital world. It is no longer about us merely shifting digital information on a confined 2D screen; instead, we now navigate within the digital realm, and the information responds directly to our actions. Imagine a couple visiting an architect to review the first draft of their new home. Now imagine them trying to navigate a 3D model using a CAD program on the 2D screen of a graphic workstation versus them walking through and around a life-sized 3D model using virtual reality. The latter experience will be easier to navigate, and it will be easier for them to draw insights and conclusions. They can relate the experience to their bodies as a known frame of reference, and it will be easier for them to express their ideas and expectations. Once the basic use of virtual reality technology is understood, the interaction with virtual digital content can be easier and of a higher quality than with traditional digital media.

The large hands of the homunculus tell us that tactile feedback is highly important. Consequently, research on haptics and tactile stimulation is far more active than that on olfactory or gustatory stimulation. However, artificial tactile stimulation for the hands is typically quite basic and faces several problems. In the physical world, stimulating the skin by exploring the surface structures of a piece of furniture or assessing the ripeness of a fruit while turning it between our fingers involves some forces while the fingers are pressing against the surface. The artificial generation of forces is, however, difficult to scale to general use cases. Advanced solutions require complex exoskeletons and are, for example, being assessed for the teleoperation of robots in space – which tells us something about their price tag and the return on investment. There are some affordable tactile solutions for the consumer market, such as tactile vests. Yet, looking at the marketing material, manufacturers apparently did not have new social interactions such as telehugging in their minds but were focusing on more violent encounters. The newly released Sony PlayStation VR2 system at least embraces haptic feedback in the headset as well as on the controllers and the trigger.

In the movie The Matrix, the character Cypher decides to betray reality and his friends to stay in the Matrix with the words: “I know this steak does not exist. I know that when I put it in my mouth, the Matrix is telling my brain that it is juicy and delicious […] Ignorance is bliss!” If he were around today, Cypher would stay with his pals in the physical world. So-called taste display technologies are rather awkward devices that, for example, use physiological effects such as galvanic taste stimulation or chemical substances to elicit responses to basic tastes in our gustatory sensors. An early device called Food Simulator was specifically designed to provide a haptic interface for biting force simulation. The researchers claimed: “Taste is the last frontier of virtual reality” and demoed their system at a virtual reality conference in 2003. Their device was a complex mechanical construct. Users had to put it into their mouths to experience biting on cheese crackers. It simulated the teeth entering softly into the cheese, making contact with the hard surface of the cracker, and biting down more firmly until, in a sudden and satisfying moment, the cracker broke, and the tension resolved. The system was combined with an injection pump delivering a mix of chemical substances to stimulate the basic tastes like sweet or umami and attempted to provide comprehensive sensory coverage for a very particular sensation. While several such technological approaches exist, even 20 years later, they are still far away from the fidelity of general-purpose display devices available for the visual or auditory domains.

The good news is that we will adapt to the metaverse quite quickly. The mapping of brain functions evolved in and for the physical world may adapt to the new sensory offerings provided by the metaverse. Lanier, one of the VR pioneers of the 1980s, reported on early work in virtual reality, in which people were given the weirdest bodies to control as their avatars. Much to their surprise, people were able to adapt quickly to new body schemas and were, for example, able to smoothly walk virtual worlds lobster-style. The idea of an adaptive homunculus and brain plasticity – the ability of the brain to modify its connections or re-wire itself – is reassuring but also raises questions. It is reassuring because flexibility helps people overcome the shortcomings and glitches of current interaction technologies, for example, learning to cope with the loss of limbs or senses in the actual world. They will quickly learn to map their actions to the capabilities offered by technology, and – assuming human-computer-interaction designers do their jobs properly – the technology itself will soon fade into the background of their conscious minds. But can we afford to have this strong focus on vision and sound in the long run? Might augmented reality be a better option for now?

Virtual reality is a highly interesting medium for many industries. Its major benefits are realized when the products or services being sold have a manifestation in the physical world that can be visually inspected and explored. If the traditional ways of selling products incorporated showrooms or brickand-mortar stores in the past, virtual reality will be an important stepstone in the digital transformation of the customer journey. However, if consumers’ decisions depend on other senses as well, augmented reality seems a viable alternative. It works in combination with physical objects so that no simulation of taste, smell or touch is required. If the scent of the interior of new cars or fresh bread will still push consumers over any decision threshold, augmented reality will be the better choice today. From the perspective of our sensory homunculus, there is a clear preference as well: augmenting the smell and touch of a juicy orange with visual information about the sustainability of its supply chain on an augmented reality device would satisfy the developed senses and the striving for more information in a consistent manner. Consequently, virtual reality is not yet ready for all our senses, but maybe research will eventually provide us with breakthrough technologies. Currently, mixed reality applications that combine physical with digital experiences are the better choice to satisfy more sophisticated sensory expectations.

Lanier, J. (2006). Homuncular Flexibility. What is your dangerous idea? Edge. https://www.edge.org/response-detail/11182

Iwata, H., Yano, H., Uemura, T., & Moriya, T. (2003). Food Simulator. ICAT 2003. icat.vrsj.org/papers/2003/00876_00000.pdf