As our modes of communication and entertainment evolve and achieve new heights, higher degrees of immersion and interactivity must be accessible to a broader audience, regardless of their device capabilities and requirements. And while audio stimulates our sense of hearing and video engages our sense of sight, advancements in haptic technology are set to add another dimension to the immersive video experience. Haptic technologies offer exciting possibilities for realism in virtual and immersive environments and can transcend the limitations of today’s devices and screens, thereby elevating and enhancing the connected experience. The alliance of pervasive connectivity (in the form of 5G) and emerging immersive technologies can support new experiences from in-built haptic feedback within VR headsets, to myriad devices enabled with tactile sensations to be felt by the user. Consequently, a more captivating and authentic virtual experience is achieved through this novel type of sensory augmentation.
But the pathway towards immersive media begins with a significant amount of research into new media codecs and standards. And this is particularly relevant in relation to haptic media coding and delivery: without the important and fundamental work within standards to first simplify development, and then support the delivery of immersive and haptic enhanced content across devices worldwide, any aspirations to deliver new depths and sensations to video-based experiences will remain a pipe dream.
Unlocking new experiences with standards
In simple terms, immersive media is three-dimensional and delivers to the user one view per eye, giving the perception of a sense of depth while encompassing user movement to encourage higher levels of immersivity.
The key to all of this lies in immersive video codecs. They allow for the delivery and consumption of data-rich video in a way that is both efficient and seamless. Without them the ubiquitous 2D—and now increasingly common 3D—volumetric and haptic-enhanced video that many users enjoy would not be possible.
Up until now, MPEG (the working group within the ISO and the IEC that develops standards for encoding audio, video and multimedia) has been a key enabler of content creation and service delivery as it plays a decisive role in describing methods for encoding, encapsulating, transmitting and decoding immersive and haptic data. In June 2021, MPEG issued a call for proposal to establish a reference architecture for future haptic standards, promoting haptics as a recognised media type in ISO base media file formats like 3gp and mp4 and streaming formats like DASH. The integration of haptic data alongside audio and video, particularly with a view to immersive content, is what is making consistent and reliable sensory rich experiences a reality.
Combating fragmentation: growth through collaboration on standardisation
Today, with more than 4.1 billion devices equipped with haptic actuators and the ability to convey haptic feedback, standards are foundational to defining the mechanisms and codecs used to deliver the sensory experience. For this reason, it is necessary to cultivate and expand the standards ecosystem to evolve simple haptic-enhanced content into more complex and immersive experiences at scale.
As it stands, the haptic environment is fragmented making it a real challenge for content creators and product developers to navigate different proprietary methods to communicate tactile sensations. This in turn drives up developer costs and lowers adoption rates. And as with audio and visual, standards in haptics are essential to ensure that all aspects involved in the end-to-end delivery of the technology are compatible.
Despite this impediment however, an industry consensus is emerging on the importance of interoperability between devices. And standards allow the industry to define and develop the fundamental methods for capturing, encoding, transmitting, decoding and replicating immersive haptic experiences.
Further, interoperability between platforms and vendors across the immersive and haptics ecosystem can only be achieved through standards. Immersive content creation, its distribution and then the decoding characteristics recommended for haptic-enabled devices depend on standards to ensure that all aspects involved in the end-to-end delivery of haptics are efficiently coded and compatible. Naturally, the genesis of such standards results from the close collaboration of pre-standards groups and industry forums whose stated mission is to limit the market fragmentation and investment stagnation that would surely arise from disparate and proprietary haptic methods that are not interoperable.
Delivering immersive experiences, today
Much industry-wide work has been undertaken to further the video processing solutions that create outstanding immersive video experiences. For example, the MPEG Visual Volumetric Video-based Coding (V3C) standard enables the mechanism for coding and streaming volumetric content through two main codecs: Video-based Point Cloud Compression (V-PCC) and MPEG Immersive Video (MIV). Both codecs are instrumental in enabling immersive-enhanced experiences in support of telelearning, entertainment, and increasingly in the delivery of live sports and esports, thereby guaranteeing that content can be consumed efficiently and seamlessly at scale across a diverse range of devices.
Meanwhile, the MPEG-I Haptics standard has been defined to create, stream, and render haptic content at scale, and has the longer-term objective of translating and distributing large scale haptic-enhanced immersive video experiences to audiences with varying device capabilities and requirements. The standard represents an important step towards maturing haptics as a media type, allowing it to reach its true potential as a first-order media type.
The future of immersive entertainment
Today, the opportunity of immersive entertainment is gaining traction, particularly as consumer electronics with in-built haptic capabilities become ever more ubiquitous. Haptic technology alone may have the potential to re-shape user’s digital interactions, making them more intuitive and tactile. As the industry steps up its standardisation efforts and the virtues of greater interoperability become ever clearer, new alliances are being forged to overcome the problem of a fragmented device ecosystem. Without standards, the industry will find itself unable to realise the essential methods for capturing, encoding, transmitting, decoding and replicating immersive haptic experiences. They open up the possibility to achieve a deeper level of engagement with content through immersive visuals and pave the way for a future where touch becomes a fundamental aspect of the digital experience.
