Filters and Effects

Overview

The Vonage Video Client SDKs allow you to use filters and effects when publishing video.

This how-to will go over:

  • Publishing a MediaStreamTrack object and applying effects
  • Applying a background blur or background replacement filter
  • Applying the advanced noise suppression filter
  • Using the Vonage Media Processor library to apply custom transformations

Publishing a MediaStreamTrack Object and Applying Effects

You can use an audio MediaStream track or a video MediaStream track as the source audio or video for a published stream. Using this feature, you can apply filters and effects, such as background blur or background replacement, to the published audio or video.

You can use the OT.getUserMedia() method to get a reference to a MediaStream that uses the camera selected by the user. You can then use the video MediaStreamTrack obtained from that MediaStream object as the source for a Video element. You can then add that Video element to an HTML Canvas element, apply filters or effects to the canvas, and use the filtered video MediaStreamTrack object obtained from the canvas as the video source for a published stream. For an example, see the Stream-Filter sample opentok-web-samples repo on GitHub.

You can use the OT.getUserMedia() method to get a reference to a MediaStream that uses the microphone selected by the user. You can then use the audio MediaStreamTrack obtained from that MediaStream object as the as the audioSource when calling OT.initPublisher(). You can then create an AudioContext object and call its createMediaStreamSource() object, passing in the MediaStream object to create a MediaStreamAudioSourceNode object. You can then apply audio filters to the MediaStreamAudioSourceNode object, which will result in the filters being applied to the published stream.

Applying a Background Blur or Background Replacement Filter

Use the OTPublisher.setVideoTransformers() method to apply a video filter for published video. You can apply a background blur filter or a background image filter.

Important: In version 2.28.0+ of the Vonage Video React Native SDK, you need to install the Vonage Media Library separately from the Vonage Video React Native SDK. For details, see Vonage Media Library integration.

The following code applies a background blur filter to a publisher's video:

<OTPublisher
  eventHandlers={{
    streamCreated: () => {
      this.publisher.setVideoTransformers([
        {
          name: 'BackgroundBlur',
          properties: JSON.stringify({
            radius: 'low',
          }),
        }
      ])
    }
  }}
  ref={instance => {
    this.publisher = instance;
  }}
/>

Note that the OTPublisher.setVideoTransformers() method takes an array of transformer objects. Each object has two properties:

A name property, set to either 'BackgroundBlur' or 'BackgroundReplacement'.

A properties property, defining options for the filter.

For details, see the documentation for the OTPublisher.setVideoTransformers() method.

The following code applies a background replacement filter to a publisher's video (supported on iOS only):

<OTPublisher
  eventHandlers={{
    streamCreated: () => {
      this.publisher.setVideoTransformers([
        {
          name: 'BackgroundReplacement',
          properties: JSON.stringify({
            image_file_path: 'path/to/image',
          }),
        }
      ])
    }
  }}
  ref={instance => {
    this.publisher = instance;
  }}
/>

To remove video filters from the published stream, call the OTPublisher.setVideoTransformers() method and pass in an empty array.

Important: Background blur and background replacement filters are resource-intensive and require devices with high processing power. It is recommended to only use these transformations on supported devices. See the following documentation: for iOS and Android

Applying the Advanced Noise Suppression Filter

Use the OTPublisher.setAudioTransformers() method to apply an audio transformer to published video. One transformer, the noise suppression filter, is supported.

Important: The noise suppression filter uses the Vonage Media Library. You need to install the Vonage Media Library separately from the Vonage Video React Native SDK. For details, see Vonage Media Library integration.

The following code applies a noise suppression filter to a publisher's video:

<OTPublisher
  eventHandlers={{
    streamCreated: () => {
      this.publisher.setAudioTransformers([
        {
          name: 'NoiseSuppression',
          properties: '',
        }
      ])
    }
  }}
  ref={instance => {
    this.publisher = instance;
  }}
/>

To remove audio transformers from the published stream, call the OTPublisher.setAudioTransformers() method and pass in an empty array.

Important: Audio transformers are resource-intensive and require devices with high processing power. It is recommended to only use these transformations on supported devices. See the following documentation:

Using the Vonage Media Processor Library To Apply Custom Transformations

See the Using the Vonage Media Processor developer guide.