Video codecs

The Vonage platform leverages the WebRTC protocol and the real-time video codecs that are supported by WebRTC. In particular, the Vonage platform supports the VP8, VP9, and H.264 video codecs.

Streams use the VP8 codec, which is supported on all clients, unless you set a preferred codec for the application that the session belongs to (see Setting the preferred video codec for a application.)

Across the broad ecosystem of devices and browsers that are supported there are varying levels of support for the VP8, VP9 and H.264 real-time video codecs.

Some endpoints support all the video codecs, and some support just one video codec. Depending on the type of application you are building and the types of browsers and devices your end users will use, your choice of preferred codec will change.

What is a video codec?

A video codec has two parts, an encoder and a decoder. It can to encode (compress) incoming digital video frames from a webcam into a stream of binary data that can be sent over a network.

It can also ingest a stream of binary data and decode (decompress) it into a flow of raw video frames that can be displayed on a screen.

The mechanism for encoding and decoding the video is the codec standard and for the purpose of this page we are going to talk about two popular ones, VP8 and H.264.

VP8 vs H.264

The VP8 real-time video codec is a software codec. It can work well at lower bitrates and is a mature video codec in the context of WebRTC.

As a software codec it can be instantiated as many times as is needed by the application within the limits of memory and CPU. The VP8 codec supports the Scalable Video feature, which means it works well in large sessions with supported browsers and devices.

The H.264 real-time video codec is available in both hardware and software forms depending on the device.

It is a relatively new codec in the context of WebRTC, although it has a long history for streaming movies and video clips over the internet.

Hardware codec support means that the core CPU of the device doesn’t have to work as hard to process the video, resulting in reduced CPU load.

The number of hardware instances is device-dependent with iOS having the best support.

Given that H.264 is a new codec for WebRTC and each device may have a different implementation, the quality can vary.

As such, H.264 may not perform as well at lower bit-rates when compared to VP8. H.264 is not well suited to large sessions since it does not support the Scalable Video feature.

VP9 vs VP8

The VP9 real-time codec is a relatively new codec. It is gaining more support among browser and hardware vendors. Despite increasing support, it still lacks the ubiquity of the other codecs.

The VP9 codec, like VP8, works well at lower bitrates than H.264. Moreover, VP9 requires even less bandwidth than VP8 to encode video. However, the improvements in compression comes with a tradeoff: encoding/decoding requires more CPU than VP8.

The VP9 codec supports Scalable Video Coding in routed sessions. For more information see VP9 Scalable Video Coding for routed sessions.

Setting the preferred video codec for a application

You can set the preferred video codec for an application in the Vonage Dashboard by selecting Application, then navigating to Video Capabilities, and finally accessing the Advanced Settings section. Valid choices are VP8, H264, or VP9, but applicability may vary depending on the type of session. By default, all projects are pre-configured with VP8 as preferred video codec.

In routed sessions (sessions that use the Media Router), the OpenTok Media Router), the publisher will negotiate with the OpenTok Media Router the codec to use, prioritizing the user-selected codec when supported by the device.

In relayed sessions, where clients send streams directly to one another, the OpenTok Media Router is not involved in the negotiation. Each publishing-subscribing pair tries to find a common video codec, preferring the user-selected codec when supported by both devices.

As a result of the negotiation, the video codec used by the pair may differ from the preferred codec set for the project.

Setting the preferred video codec through the SDK

The Publisher Preferred Video Codec API gives you fine-grained control over how video codecs are prioritized during session negotiation. With this API, you can:

• Specify a custom priority list of codecs to influence negotiation.
• Delegate prioritization to the SDK in automatic mode.
• Rely on defaults from your project settings if you don't set anything.

Web SDK

export type VideoCodec = 'vp8' | 'vp9' | 'h264';
export type PreferredVideoCodecs = 'automatic' | [VideoCodec, ...VideoCodec[]];

// Default (omitted)
OT.initPublisher('div', {});

// Automatic mode
OT.initPublisher('div', {
  preferredVideoCodecs: 'automatic'
});

// Manual priority order
OT.initPublisher('div', {
  preferredVideoCodecs: ['vp9', 'vp8']
});

// Invalid usage (empty list)
OT.initPublisher('div', {
  preferredVideoCodecs: []
}, function (err) {
  if (err) {
    console.error(err.name);    // 'OT_INVALID_PARAMETER'
    console.error(err.message); // 'preferredVideoCodecs must be "automatic" or a non-empty array of valid codec strings'
  }
});

iOS SDK

// Manual mode
OTVideoCodecPreference *pref = [OTVideoCodecPreference manualWithCodecs:@[
    @(OTVideoCodecTypeVP9),
    @(OTVideoCodecTypeH264),
    @(OTVideoCodecTypeVP8)
]];
OTPublisherKitSettings *settings = [[OTPublisherKitSettings alloc] init];
settings.videoCodecPreference = pref;

// Automatic mode
OTVideoCodecPreference *pref = [OTVideoCodecPreference automatic];
OTPublisherKitSettings *settings = [[OTPublisherKitSettings alloc] init];
settings.videoCodecPreference = pref;

Android SDK

// Manual mode
PublisherKit.PreferredVideoCodecs preferredVideoCodecs =
    PublisherKit.PreferredVideoCodecs.manual(
        new ArrayList<PublisherKit.PreferredVideoCodecs.Codec>(
            List.of(PublisherKit.PreferredVideoCodecs.Codec.VP9,
                    PublisherKit.PreferredVideoCodecs.Codec.H264)));

Publisher publisher = new Publisher.Builder(MainActivity.this)
    .preferredVideoCodecs(preferredVideoCodecs)
    .build();

// Automatic mode
PublisherKit.PreferredVideoCodecs preferredVideoCodecs =
    PublisherKit.PreferredVideoCodecs.automatic();

Publisher publisher = new Publisher.Builder(MainActivity.this)
    .preferredVideoCodecs(preferredVideoCodecs)
    .build();

Windows SDK

// Manual mode
var pref = new PreferredVideoCodecs(new List<PreferredVideoCodecs.Codec> {
    PreferredVideoCodecs.Codec.VP9,
    PreferredVideoCodecs.Codec.H264,
    PreferredVideoCodecs.Codec.VP8
});
builder.PreferredVideoCodecs = pref;

// Automatic mode
builder.PreferredVideoCodecs = PreferredVideoCodecs.Automatic();

// Default (uses project settings)
builder.PreferredVideoCodecs = null;

// Invalid (throws)
try {
    builder.PreferredVideoCodecs = new PreferredVideoCodecs(new List<PreferredVideoCodecs.Codec>());
} catch (ArgumentException ex) {
    // "Must specify at least one codec"
}

Linux and macOS SDK

// Manual mode: specify an ordered priority list of codecs
otc_publisher_settings* publisher_settings = otc_publisher_settings_new();

otc_video_codec_type codecs[] = {
    OTC_VIDEO_CODEC_VP9,
    OTC_VIDEO_CODEC_H264,
    OTC_VIDEO_CODEC_VP8
};

otc_status status = otc_publisher_settings_set_preferred_video_codecs(
    publisher_settings,
    codecs,
    sizeof(codecs) / sizeof(codecs[0])
);

otc_publisher_callbacks publisher_callbacks = {0};
struct otc_publisher* publisher =
    otc_publisher_new_with_settings(&publisher_callbacks, publisher_settings);

// Done with settings object
otc_publisher_settings_delete(publisher_settings);

// Automatic mode: let the SDK choose priority automatically
otc_publisher_settings* publisher_settings = otc_publisher_settings_new();

otc_status status = otc_publisher_settings_set_preferred_video_codecs_automatic(
    publisher_settings
);

otc_publisher_callbacks publisher_callbacks = {0};
struct otc_publisher* publisher =
    otc_publisher_new_with_settings(&publisher_callbacks, publisher_settings);

// Done with settings object
otc_publisher_settings_delete(publisher_settings);

It's important to understand that the final video codec could vary from the preferred codec depending on the outcome of the negotiation phase.

Codec coverage

The following tables list the real-time video codec capabilities of the supported endpoints.

¹ VP8 is available in Safari 12.1+, which ships on macOS 10.14.4 and is also available for macOS 10.13.6 and 10.12.6.

² VP9 is available in Safari 15+ and is only supported on devices with hardware decoding (such as the M3 Macbook Pro).

¹ Chrome on Android only supports H.264 on devices that contain the Qualcomm and Exynos chipsets and require Chrome 65 or higher.

² VP8 is available in Safari on iOS 12.2+.

³ VP9 is available in Safari 15+ and is only supported on devices with hardware decoding (such as the iPhone 15 Pro).

⁴ VP9 is available in iOS 17+ and is only supported on devices with hardware decoding (such as the iPhone 15 Pro).

⁵ WebView Android only supports H.264 on devices that contain the Qualcomm and Exynos chipsets and require Chrome 65 or higher.

⁶ VP8 is available on iOS 12.2+.

⁷ VP9 is available on iOS 15+ and is only supported on devices with hardware decoding (such as the iPhone 15 Pro).

* Hardware H.264 is supported on devices that contain the Qualcomm and Exynos chipsets, in addition to a subset of HiSilicon and MediaTek chipsets. Fallback to software H.264 is supported on Android M or higher.

** VP9 support requires 2.27+ of the Vonage Video native client SDKs (for Android, iOS, macOS, Linux, and Windows).

Note: H264 support requires version 2.29+ of the Vonage Video client SDK for macOS.

Codec feature support across endpoints

* Percentage of Android devices that support the codec

** VP9 is supported on Safari 15+ (desktop and iOS) and WebView on iOS 15+, but only on supported devices with hardware encoding, like the M3 Macbook and the iPhone 15 Pro.

*** VP9 is supported on Firefox, but scalable video is not. Hardware and software scalable video support is not supported on all devices.

Detecting codec support in clients

Not all Android devices support the H.264 codec, and older versions of Safari do not support VP8. OpenTok.js and the Android SDK include methods for checking the supported codecs available to the client.

Detecting supported codecs in the browser

The OT.getSupportedCodecs() method returns a Promise that is resolved (on success) with an object that has two properties: videoDecoders, an array of supported video codecs for decoding, and videoEncoders, an array of supported video codecs for encoding.

The following example gets the list of supported codecs for encoding and decoding video streams:

(async () => {
  try {
    const supportedCodecs = await OT.getSupportedCodecs();
    if (supportedCodecs.videoEncoders.indexOf('H264') < 0 && supportedCodecs.videoDecoders.indexOf('H264') < 0) {
      // They do not support encoding or decoding H264 let's tell them to use a different browser
    }
  } catch(err) {
    console.log(err);
  }
})();

Detecting supported codecs using the Android SDK

• MediaUtils.SupportedCodecs.getSupportedCodecs(context)

• MediaUtils.SupportedCodecs

• MediaUtils.VideoCodecType

The MediaUtils.SupportedCodecs.getSupportedCodecs(context) returns a MediaUtils.SupportedCodecs object that that has two properties: videoDecoders, an ArrayList of supported video codecs (defined by the MediaUtils.VideoCodecType class) for decoding, and videoEncoders, an ArrayList of supported video codecs (defined by the MediaUtils.VideoCodecType class) for encoding.

The following example gets the list of supported codecs for encoding and decoding video streams:

    MediaUtils.SupportedCodecs supportedCodecs =
        MediaUtils.SupportedCodecs.getSupportedCodecs(context);
    if (supportedCodecs.videoEncoders.indexOf(MediaUtils.VideoCodecType.VIDEO_CODEC_H264) < 0
        && supportedCodecs.videoDecoders.indexOf(MediaUtils.VideoCodecType.VIDEO_CODEC_H264) < 0)
    {
        // The device does not support encoding or decoding H264.
    }

Issues to consider when selecting your preferred real-time video codec

Interoperability

The main interoperability conflict is around Android devices, older versions of Safari, and Linux. VP8 works on all Android devices, both on Android Chrome and the Android SDK, but H.264 codec support on Android is not ubiquitous. Older versions of Safari do not have VP8 codec support. The Linux SDK does not support H.264.

Session size

H.264, VP8 and VP9 can work well for endpoints in small sessions (for example, one to three participants). However, since there is no Scalable Video support with H.264, we do not recommend using this codec for large sessions.

Video quality

Since the same VP8 video codec implementation is used on almost all endpoints the quality is roughly the same. VP8 works well at lower bitrates.

In addition, Scalable Video is available with VP8. Scalable Video significantly improves the video quality in larger sessions.

The quality of H.264 will vary across devices since the implementation of H.264 varies.

In addition, we have encountered differences in quality depending on the operating system version running on the device. The quality of H.264 at lower bitrates is generally not as good as VP8.

H.264 works well on iOS devices since they have good support for H.264 hardware acceleration. This reduces the CPU load and improves battery life.

VP9 offers improved video quality over VP8 when operating at the same bitrate. The quality improvements may require increased CPU load than VP8, though.

Example scenarios

Here are the recommended video codec to use in some example scenarios:

• In sessions with 2-3 participants where interoperability with older versions of Safari is critical, use the H.264 video codec.
• In sessions with 2-3 participants where only iOS devices are involved, use the H.264 video codec to take advantage of the hardware acceleration.
• Webinars and Large Classroom sessions should use the VP8 codec to take advantage of the Scalable Video feature.
• In Sessions where support for all Android devices is critical, use the VP8 codec.