Ericsson Holographic Communication (EHC) enables fully immersive, real-time, 3D experiences that can be built with consumer-grade mobile devices over 5G, while providing a greater sense of spatial awareness, proximity and presence. In collaboration with the Vonage Video API team, they have built a demo to be able to have Holographic video calls using the Vonage Video SDK and the TrueDepth cameras on iPhones and iPads.
Initially, AR glasses were used to project holograms. Later, consumer-grade mobile devices like phones were included, allowing users to experience holographic communication on their existing mobile devices. While we believe there are countless use cases, the project as a start focuses on e-health and entertainment use cases. In the world of entertainment, you could be sitting in your living room and fully enjoy a holographic concert of your favourite singer or watch your favourite vlog in 3D.
These holographic experiences provide a sense of presence that cannot be achieved with 2D video alone, increasing engagement, bringing new ways of interacting with the content and making it more enjoyable for the viewer. In telehealth environments, in remote therapy sessions, it is important to establish a strong connection and build trust between the doctor and patient to ensure the session is effective and productive.
Attendee trying out a demo
A 3D presence in the room can help with intimacy and understanding emotional nuances. This bond makes patients feel comfortable and open up, leading to more successful outcomes. The team saw an opportunity to bring the technology to more people by creating a Unity SDK for use with the Vonage Video SDK.
Holographic video with AR glasses
To get holographic communication demo working with the Vonage Video SDK some new features were developed. Most importantly, a new WebRTC camera capturer was built to capture both depth data and standard video data simultaneously.
Example of TrueDepth Camera
The TrueDepth cameras provide a large amount of depth data. For this to be efficiently handled, the team built a library with high-performance compression algorithms. This is used to compress the depth data before it is sent to the receiver.
To send this data, a custom implementation of the WebRTC Multiplex codec was created. The Multiplex codec allows you to send more than just the usual audio and video data, and is traditionally used to send alpha data with a fixed size. The custom implementation removes this fixed alpha data and replaces it with dynamic depth data after it has been compressed. This allows for the depth and video data to be sent and received at the same time to avoid having to do any syncing between the two.
In addition to video data, depth data is also being handled. As you can imagine this would lead to an increase in the amount of bandwidth being used. Each frame is composed of compressed video and depth information. The depth buffer can support up to 350KB, but the actual amount of bandwidth needed varies depending on several factors, such as the resolution, the frames per second, how close the user is to the camera, etc. In the demo shown below, on average around 10Mbits of depth data per second were transferred. The demo application runs on a stable Wi-Fi connection and was demonstrated on a 5G stand-alone network at Mobile World Congress.
There are two flows in the demo application, the sender and the receiver. As mentioned earlier the depth data is then compressed and sent to the receiver via the Vonage Video SDK using the custom WebRTC Multiplex codec.
The receiving flow of the application uses a Holographic Video Unity SDK. The Vonage Video SDK receives the data from the senders side, which is then passed to the Unity SDK which handles the rendering of the hologram. Internally the Unity SDK is using the Unity AR Foundation library to place the hologram in the receivers world!
The demo application was exhibited during Mobile World Congress this year and received a lot of positive feedback from the attendees! The excitement was evident among the crowd. One attendee said "It feels like we're already experiencing the future today," and another asked, "When will it become the new norm?" This interest suggests that the technology could enhance efficiency, communication, and user experience across various fields, creating new use cases that we haven't even imagined yet.
Follow the Vonage Developer Experience team on X, formerly known at Twitter, or join the Vonage Community Slack channel. If you are interested in learning more about Ericsson Holographic Communication reach out to team over email.
){kind=small}
