When creating a Vonage Video publisher, the stream can be sourced directly from a user camera, from a <video> element, or a HTML <canvas> element. Once pixels get drawn to the canvas, they can be easily manipulated before being used in a Video API session.

In this tutorial, you'll learn how to remove a green screen and replace it with a new, custom image that you can include in your video calls.

Overview of the project components

Several components are required to make the project work. Firstly, a <video> element will take a stream from the user's camera. Each frame, the video element content will be drawn on a canvas, where we will loop through pixels to remove those which are green. On a second canvas, we will draw the replacement background image and then layer the first canvas' non-green pixels on top.

With our desired output on a canvas, we can use the canvas as a source for a Vonage Video API publisher, which we can use in our video sessions with friends.

If you want to look at the finished code, you can find it at https://github.com/nexmo-community/video-green-screen

Scaffold Markup

Create a new project folder followed by a new file index.html, populating this file with the following code:

<video id="v1" width="320" height="240" autoplay=""></video>
  <canvas id="c1" width="320" height="240"></canvas>
  <canvas id="c2" width="320" height="240"></canvas>
  <div id="opentok-publishers"></div>
  <div id="opentok-subscribers"></div>
  <script>
    // Create references to the video and canvas elements
    const v1 = document.getElementById('v1')
    const c1 = document.getElementById('c1')
    const c2 = document.getElementById('c2')

    // Get canvas context
    const c1Ctx = c1.getContext('2d')
    const c2Ctx = c2.getContext('2d')
  </script>

You'll also need the image that you want to replace your green screen within the project folder. This tutorial will use one of the Vonage brand gradients. After you get the canvas contexts, load the image:

const backgroundImage = new Image()
backgroundImage.src = 'vonage-gradient.png'

Get Webcam Video

Set the <video> element's source to the stream from the user's webcam. This snippet will pick the default camera:

navigator.mediaDevices.getUserMedia({ video: true })
  .then(stream => { v1.srcObject = stream })

Create an empty process() function. Once the user's video device is ready and 'playing', run the function every frame:

v1.addEventListener('play', () => {
  setInterval(process, 0)
})

function process() {

}

Draw Video Stream to the Canvas

Update process():

function process() {
  c1Ctx.drawImage(v1, 0, 0, 320, 240)
  c2Ctx.drawImage(backgroundImage, 0, 0, 320, 240)
}

Refresh your page, and you should see your <video> element, your first <canvas> with a duplicate image, and your second <canvas> with the new background image. The goal is to get any non-green pixels on top of the background in the second canvas.

Loop Through Pixels

In a canvas, the entire image represented in a single long array of pixels. While you may initially believe that our 320x240 image will have 76,800 entries in the array, you'd be mistaken.

canvas pixels

Each visible pixel is made up of four array items - one for its red value, one for green, one for blue, and the final to set its opacity. These values are important as we build and use the loop.

Get this frame's pixels array inside of the process() function, and build the loop:

const frame = c1Ctx.getImageData(0, 0, 320, 240)
const pixels = frame.data

for(let i=0; i<pixels.length; i+="4)" {="" }="" <="" code=""></pixels.length;>

Notice that the counter is set to increment by 4. Each time this loop runs, `i` will be the array index of the next visible pixel's red value. Understanding HSL ----------------- Before green pixels can be removed, I'd like to introduce you to the Hue Saturation Lightness (HSL) color format. ![Hue is a color wheel, sauturation is the amount of grey, lightness is a scale of black to white](https://d226lax1qjow5r.cloudfront.net/blog/blogposts/use-a-green-screen-in-javascript-with-vonage-video/hsl.png "Hue is a color wheel, sauturation is the amount of grey, lightness is a scale of black to white") You can think of hue as a color wheel - and use the position on the wheel to specify a color, from 0 to 360. The green 'range' might be different for each person, but 90 to 200 works well for me. However, when reading and writing pixels to a `<canvas>` you must use the Red Green Blue (RGB) color format. At the very bottom of your `<script>`, add this `RGBToHSL()` function [provided on CSS Tricks](https://css-tricks.com/converting-color-spaces-in-javascript/#rgb-to-hsl): ```blok {"type":"codeBlock","props":{"lang":"js","code":"function%20RGBToHSL(r,%20g,%20b)%20%7B%0A%20%20r%20/=%20255;%20g%20/=%20255;%20b%20/=%20255;%0A%20%20let%20cmin%20=%20Math.min(r,g,b),%20%0A%20%20%20%20%20%20cmax%20=%20Math.max(r,g,b),%20%0A%20%20%20%20%20%20delta%20=%20cmax%20-%20cmin,%20%0A%20%20%20%20%20%20h%20=%200,%20s%20=%200,%20l%20=%200;%0A%20%20if%20(delta%20==%200)%20h%20=%200;%0A%20%20%20%20else%20if%20(cmax%20==%20r)%20h%20=%20((g%20-%20b)%20/%20delta)%20%25%206;%0A%20%20%20%20else%20if%20(cmax%20==%20g)%20h%20=%20(b%20-%20r)%20/%20delta%20+%202;%0A%20%20%20%20else%20h%20=%20(r%20-%20g)%20/%20delta%20+%204;%0A%20%20h%20=%20Math.round(h%20*%2060);%0A%20%20if%20(h%20&lt;%200)%20h%20+=%20360;%0A%20%20l%20=%20(cmax%20+%20cmin)%20/%202;%0A%20%20s%20=%20delta%20==%200%20?%200%20:%20delta%20/%20(1%20-%20Math.abs(2%20*%20l%20-%201));%0A%20%20s%20=%20+(s%20*%20100).toFixed(1);%0A%20%20l%20=%20+(l%20*%20100).toFixed(1);%0A%20%20return%20%5Bh,%20s,%20l%5D%0A%7D%0A"}} ``` Making Green Pixels Transparent ------------------------------- Inside the `process()` loop, get the RGB and HSL values for each pixel, and set pixels which are green to be transparent: ```blok {"type":"codeBlock","props":{"lang":"js","code":"const%20%5Br,%20g,%20b%5D%20=%20%5Bpixels%5Bi%5D,%20pixels%5Bi+1%5D,%20pixels%5Bi+2%5D%5D%0Aconst%20%5Bh,%20s,%20l%5D%20=%20RGBToHSL(r,%20g,%20b)%0A%0Aif(h%20&gt;%2090%20&amp;&amp;%20h%20&lt;%20200)%20%7B%0A%20%20pixels%5Bi+3%5D%20=%200%0A%7D%0A"}} ``` After the loop, update the canvas image: ```blok {"type":"codeBlock","props":{"lang":"js","code":"frame.data%20=%20pixels%0Ac1Ctx.putImageData(frame,%200,%200)%0A"}} ``` You may find that `90` and `200` needs updating, given the color of your screen and lighting. ![The first canvas has no background - appearing white](https://d226lax1qjow5r.cloudfront.net/blog/blogposts/use-a-green-screen-in-javascript-with-vonage-video/removing-green.png "The first canvas has no background - appearing white") Draw Remaining Pixels on Replacement Background ----------------------------------------------- For each pixel that is remaining in the first canvas, draw it on the second. After the `if` statement in the `process()` loop, add an `else` condition: ```blok {"type":"codeBlock","props":{"lang":"js","code":"if(h%20&gt;%2090%20&amp;&amp;%20h%20&lt;%20200)%20%7B%0A%20%20pixels%5Bi+3%5D%20=%200%0A%7D%20else%20%7B%0A%20%20c2Ctx.fillStyle%20=%20%60rgba($%7Br%7D,%20$%7Bg%7D,%20$%7Bb%7D,%201)%60%0A%20%20const%20x%20=%20(i/4)%20%25%20320%0A%20%20const%20y%20=%20Math.floor((i%20/%204)%20/%20320)%0A%20%20c2Ctx.fillRect(x,%20y,%201,%201)%0A%7D%0A"}} ``` The `x` and `y` values are the visual pixels, so the `i` value should be divided by 4. ![The second canvas now has the non-removed pixels](https://d226lax1qjow5r.cloudfront.net/blog/blogposts/use-a-green-screen-in-javascript-with-vonage-video/replacement-bg.png "The second canvas now has the non-removed pixels") Include Canvas in Video API Session ----------------------------------- Create a new project in your [Vonage Video Dashboard](https://tokbox.com/account). Once created, scroll down to Project Tools and create a new Routed session. Take the Session ID and create a new token. At the top of your `<script>`, create three new variables with data from the project dashboard: ```blok {"type":"codeBlock","props":{"lang":"js","code":"const%20sessionId%20=%20'YOUR_SESSION_ID'%0Aconst%20apiKey%20=%20'YOUR_PROJECT_API_KEY'%0Aconst%20token%20=%20'YOUR_TOKEN'%0A"}} ``` Next, copy the `<script>` tag from the [Vonage Video API Client SDK page](https://tokbox.com/developer/sdks/js/#loading) and put it above your existing `<script>` tag. At the bottom of your `<script>` tag, get your basic Vonage Video API session initialized and publish from the second canvas: ```blok {"type":"codeBlock","props":{"lang":"js","code":"//%20Initialize%20session%0Aconst%20session%20=%20OT.initSession(apiKey,%20sessionId)%0A%0A//%20Create%20publisher%0Aconst%20publisher%20=%20OT.initPublisher(%22opentok-publishers%22,%20%7B%0A%20%20videoSource:%20c2.captureStream().getVideoTracks()%5B0%5D,%0A%20%20width:%20320,%0A%20%20height:%20240%0A%7D)%0A%0A//%20Once%20connected%20to%20session,%20publish%20the%20publisher%0Asession.connect(token,%20()%20=&gt;%20%7B%0A%20%20session.publish(publisher)%0A%7D)%0A%0A//%20Show%20other%20users'%20streams%0Asession.on('streamCreated',%20event%20=&gt;%20%7B%0A%20%20session.subscribe(event.stream,%20%22opentok-subscribers%22)%0A%7D)%0A"}} ``` Hide Elements ------------- The `<video>` and `<canvas>` elements are required to make this work, but you probably don't want them visible in your webpage. In your `<head>`, add the following CSS to hide them: ```blok {"type":"codeBlock","props":{"lang":"html","code":"%3Cstyle%3E%0A%20%20#v1,%20#c1,%20#c2%20%7B%20display:%20none%20%7D%0A%3C/style%3E%0A"}} ``` What Will Your Background Be? ----------------------------- Hopefully, you found this blog post useful and can now create custom backgrounds to your heart's content. While we focused on greenscreens, any pixel-level manipulation can be done with the same approach. To take this further, you may choose to provide users with controls that alter the HSL values which are 'in range' to be replaced, or a file selector to change the image. You can find the final project at [https://github.com/nexmo-community/video-green-screen](https://github.com/nexmo-community/video-green-screen) As ever, if you need any support feel free to reach out in the [Vonage Developer Community Slack](/community/slack). We hope to see you there.