Video manipulation – What is happening?

In my project for this week, the setup function shows the image captured from the camera and asks for a color to search for (the color can be selected by clicking on some point of the image), then the program displays pixels of a pre-recorded video instead of all the pixels of the image captured from the camera that are of that color (and within a threshold), while the rest of the image turns to random noise (colors). I feel like the computer is having a hard time finding the color I want it to look for (and similar ones to that color), and is looking for colors of similar darkness instead. I’ll try working on this but I like what is happening anyway.

 

Here are the code and the video files I used for my project:

https://drive.google.com/a/nyu.edu/folderview?id=0BwGOKPbccFXuVGV5OWxmR05TODA&usp=sharing

(For testing: I find it best to click on my mouth or some sort of middle-dark, possibly pinkish color, that gives the best results.)