Post-screen interfaces: When the environment becomes the interface

Interaction is becoming multimodal

For interactions to become multimodal – meaning that input is no longer limited to the mouse, keyboard and screen, but also includes gestures, voice and facial expressions – it is not individual innovations that are required, but the interplay of several technological developments. What matters is not each technology in isolation, but the synergies that arise from them. Language plays a central role. Voice-based interaction is one of the most natural forms of human communication. Digital systems can answer questions, perform tasks or provide information without the need for a display. With the integration of generative AI, these systems are increasingly evolving into conversational interfaces that understand context and coordinate more complex tasks – a development path closely linked to the concept of AI agents within the AI Empowerment cluster.

At the same time, the body is also gaining importance as an interface. Cameras and sensors detect hand movements, gaze directions or positions in space and translate these signals into digital actions. In mixed-reality systems, for example, users can select a virtual object simply by looking at it and confirm an action with a small hand movement. Digital content no longer appears on a separate screen, but directly within the space.

Experimental neurointerfaces represent a particularly interesting development. The AlterEgo system from the MIT Media Lab, for example, enables users to communicate with computers without speaking or typing. The wearable device measures minute neuromuscular signals in the jaw area that are generated when people ‘speak’ words internally. An AI translates these signals into digital commands or text. This creates a form of virtually silent human-computer communication, in which information can be retrieved or devices controlled without the use of a visible interface. This opens up new forms of interaction with digital agents that are both more inclusive and function discreetly in public spaces.

These examples illustrate a fundamental shift. Interaction no longer takes place exclusively via a single input method. Instead, modern systems combine several forms of interaction simultaneously. Speech, gestures, gaze direction and contextual information interlock to create a multimodal logic that enables more flexible communication between humans and machines. 

Physical space itself is also increasingly becoming part of digital interfaces. Systems detect position, depth and movement and can position digital content accordingly. Augmented reality applications already demonstrate this principle today. For example, a virtual piece of furniture can appear in your own living room, or a digital model can be placed on a table.

At the same time, interfaces are emerging that function entirely without a traditional user interface. Information is not displayed permanently, but appears contextually. Light, sound or projections can serve as an interface.

One example of such ambient interfaces is smart lighting that reacts to emotions. Cameras or sensors analyse facial expressions, voice or biometric signals and detect stress or aggression. The system can respond by automatically adjusting light colour, brightness or atmosphere. The aim of such systems is to defuse escalating situations at an early stage, for example in schools, public spaces or transport systems. The environment itself thus becomes an active interface that reacts to human emotions.

Overall, this is shifting digital interaction towards a more context-based approach: users no longer need to actively operate systems to the same extent. Instead, systems respond more strongly to the environment, behaviour and situation, and technology is distributed across various devices and sensors. Interaction thus arises from the interplay between space, devices and users.

However, such interfaces must be designed responsibly. Issues of data protection, reliability and inclusive design will be decisive in determining how widely these technologies actually become established. For in order to react contextually, the context must be understood. This raises the question of how far we are willing to allow tech companies to intrude on our privacy. Systems that accompany us in our daily lives must be trustworthy and respect our privacy. The macro-trend ‘Trusted Ecosystems’ explores in greater detail how data sovereignty can be combined with technological openness.

New touchpoints for media, brands and retail

Many elements of post-screen interfaces are already visible on the market. They rarely appear in isolation. In most cases, several technologies combine to create new usage scenarios.

One failed – but popular – example from the past is the so-called AI Pin: a small wearable device that acted as a personal AI assistant and was attached directly to clothing. These devices usually operated without a traditional screen and were controlled via voice, gestures or contextual data. AI models handled tasks such as retrieving information, sending messages or recognising objects in the surroundings.

The Humane AI Pin was one of the best-known early prototypes in this device category. The aim of such devices is to transfer many of the smartphone’s functions to an invisible, constantly available interface.

In the case of the AI Pin, the company failed to build a sustainable user base, and so sales of the product were discontinued after less than a year. The Pin failed to deliver on its performance promises, and the hardware was also disappointing due to regular overheating. Added to this was the fact that the price of around 700 US dollars was very high for a device with limited functionality.

Thus, the first generation of such systems also reveals their limitations: the devices face the challenge not only of replicating existing smartphone functions, but of offering clear added value. Without a tangible advantage over established devices, the use of additional hardware remains unattractive to many users. The success of such interfaces therefore depends largely on whether they actually simplify or speed up interaction, or open up new, context-based usage scenarios.

Spatial interfaces are also evolving, such as Meta’s Surface Keyboard for the Meta Quest 3. The system transforms any flat surface, such as a table or desk, into a virtual keyboard and touchpad. The VR headset recognises the geometry of the surface and displays a digital keyboard exactly where the user’s hands are positioned. This allows users to type as they would on a real keyboard or navigate using gestures. The physical space itself thus becomes the interface. Such developments demonstrate how digital interaction is increasingly emerging from the combination of space, body and context.

For the retail sector, this gives rise to new forms of shopping experiences. Customers can view products in augmented reality, access information or experience digital configurations directly within the space.

In the media sector, too, the logic of attention is changing. Content no longer appears exclusively on screens. Audio formats, wearables, spatial interfaces or connected devices are creating new ways of delivering content. For example, a system can recognise the situation a user is in and adapt content accordingly. A message is read aloud because the user is currently on the move. Visual information only appears when a screen or spatial interface is available.

For media companies and marketers, this means an expansion of the available touchpoints. Attention is no longer generated solely at the moment of screen contact. It can also arise through interactions with digital systems, via spatial interfaces or through smart environments.

Outdoor advertising can also change in this context. Digital advertising spaces remain physically present but can be enhanced with digital layers. Users view additional content via smartphones or AR glasses. The physical space thus becomes the starting point for an enhanced digital experience.

At the same time, the boundaries between media use and transactions are blurring. An interaction with an AI system can lead directly to an order. A spatial AR experience can be directly linked to a purchase.

As a result, media, commerce and services are converging more closely. Interfaces are becoming touchpoints where information, inspiration and transactions are directly linked – a development that can also be observed in the Smart Commerce & Media trend cluster.

What companies should be doing right now

Companies implementing post-screen interfaces should link each application to a clearly defined use case. For each use case, a specific answer should be provided as to which existing interaction is being replaced or improved, and why a screenless or multimodal interface is superior in this instance. Applications that do not offer a clearly measurable advantage over existing solutions should not be implemented. The case of the Humane AI Pin shows that new devices with no functional added value over the smartphone are unlikely to generate sustained usage.

For context-based interaction, companies must tailor their data and system architecture specifically to this purpose. Systems should be capable of processing situational information such as location, usage context or intention in real time. At the same time, clear rules should be defined regarding which data may be collected, how long it is stored and who has access to it. Users must be able to understand at all times why a system displays certain information or performs an action.

Post-screen interfaces should be technically designed to work seamlessly with existing touchpoints. Each application should offer at least two alternative modes of interaction. For example, voice and visual output, or audio and display. Organisations should avoid creating entirely new, isolated systems that are not integrated into existing ecosystems.

Content and services must also be designed specifically for these interfaces. For each use case, it should be defined what information is required at what moment and in what form it is presented. Content should be deliberately streamlined, context-specific and action-oriented. Companies should avoid simply adopting existing content unchanged, and instead develop their own formats for audio, spatial interfaces or situational interaction.

Conclusion

A new approach to digital interaction

Post-screen interfaces do not represent an abrupt replacement of the screen. Rather, a new form of digital interaction is emerging, in which the screen is merely one of several possible interfaces.

Technology is distributed across devices, spaces and sensors. Interaction arises from the interplay of speech, gestures, spatial perception and contextual information. As a result, interfaces become less visible, yet at the same time are more deeply integrated into everyday life.

For media, platforms and brands, this changes the logic of digital touchpoints. Attention is no longer generated exclusively where a display is visible, but can also arise in conversations with digital systems, in spatial interfaces or in smart environments.

The screen is not disappearing. However, its role is changing fundamentally. Instead of forming the centre of digital interaction, it is becoming part of a larger network of interfaces.

Digital systems are thus moving towards an environment in which technology is perceived less as a tool and functions more as invisible infrastructure in the background. The world itself is increasingly becoming the interface.

The media content in this blog post was created using AI.