Detecting video changes

Tao3D can capture live video, and makes it quite easy to perform image analysis using shaders. Here is a simple example that shows how to highlight moving parts in a video by colorizing them in red and blue.

 

Capturing live video

Tao3D can capture live video in a texture by using the VLCAudioVideo module and using the qtcapture:// URL on MacOSX or dshow:// on Windows (I am not sure what the right URL would be on Linux).

So below is a simple program that would capture a live video stream and display it at the center of the screen:

import VLCAudioVideo
  
WEBCAM -> if is_available mac then "qtcapture://" else "dshow://"

movie_texture WEBCAM
rectangle texture_width, texture_height

Saving a texture

If we want to detect movement, we need to save a reference frame to compare to. This can be done by using a dynamic texture in which we play the movie, but that we only update at specific times. Here is what it looks like to capture a snapshot every second:

lastTex -> 0
  
if time mod 1 < 0.1 then
    frame_texture 640, 480,
        color "white"
        movie_texture WEBCAM
        rectangle texture_width, texture_height
    lastTex := texture

The solution above is a bit crude, since it hard-codes the frame texture size. Normally, we should capture the actual frame size from the input video:

lastTex -> 0
frameWidth  -> 640
frameHeight -> 480
  
if time mod 1 < 0.1 then
    frame_texture 640, 480,
        color "white"
        movie_texture WEBCAM
        rectangle texture_width, texture_height
    lastTex := texture

Comparing textures

A simple shader program that compares two textures and highlights the difference in red/blue will look like this:

uniform sampler2D last;
uniform sampler2D cur;

void main()
{
    vec4 old = texture2D(last, gl_TexCoord[0].st);
    vec4 new = texture2D(cur, gl_TexCoord[1].st);
    gl_FragColor = vec4(0.3*vec3(old.r + old.g + old.b),1) + vec4(1,0,0,0) * (new - old);
}

The last line adds a grey image and a red channel that contains the difference between the new and old picture. Apparently, when this difference becomes negative, GLSL will push up the blue and green channels of the resulting fragment color as a result of color normalization.

I will save this program in a file called diff.fs for easier editing. Remember that Tao3D will reload the shader code when you change it.

Comparing the snapshot and the current image

We can now run the shader program to compare the snapshot and the current image from the camera:

import Slides
import VLCAudioVideo

WEBCAM -> if is_available mac then "qtcapture://" else "dshow://"

base_slide "Test",
    lastTex -> 0

    color "white"
    contents 0,
        if time mod 1  640
            frameHeight -> 480
            frame_texture frameWidth, frameHeight,
                color "white"
                movie_texture WEBCAM
                rectangle texture_width, texture_height
                frameWidth := texture_width
                frameHeight := texture_height
            lastTex := texture
        else
            texture lastTex
        rectangle -400, 0, 640, 480

    contents 0,
        texture_unit 0
        texture lastTex
        texture_unit 1
        movie_texture WEBCAM
        shader_program
            fragment_shader_file "diff.fs"
        shader_set last := 0
        shader_set cur  := 1
        rectangle 400, 0, 640, 480

The result will look something like this:

It’s of course more interesting when it helps highlight small movements:

Going further

This is only a starting point. You can explore ideas such as:

  • Using other sources such as a webcam.
  • Exploring the capabilities of shaders. The Filters module shows a few common algorithms in image processing that can serve as a starting point.
  • Storing more than one texture for finer analysis.

That was easy!

Christophe de Dinechin

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s