Background Subtraction and Models

Looked at three background models:

• Codebook
• Learn and keep ranges of values for each pixel. They will be treated as background pixels. Clear stale pixels to remove moving foregrounds from the learning period.
• Foreground objects are detected by subtracting pixels from the ranges (with some allowances).
• Mixture of Gaussians
• The intensity of each pixel is modeled by a few Gaussians. Background values are the most probable ones (highly weighted).
• Model gets continuously updated (adaptive). No separate learning stage.
• Foreground objects are detected by subtracting pixels from the most probable Gaussian.
• Color Co-occurrence:
• Learn and keep a table of color co-occurrences at each pixel location. 
• Input pixel is classified as foreground, background or moving background.
• A reference background image is maintained to compute 'background difference' after binary thresholding.
• Temporal difference from frame to frame will be updated to the co-occurrence tables.
• Foreground/(Static, Moving) Background objects are classified using Bayes decision rule.

Running the Samples with street side video:

• bgfg_codebook (CodeBook method)
• Least CPU demanding.
• Connected objects does not resemble original shape.
• Good result in default parameters.
• segment_objects (Mixture of Gaussians and connected-object refinement)
• More CPU demanding ( 50% CPU on a dual core)
• Lower background-ratio parameter gives a more solid shape of a moving car.
• alpha (learning-rate parameter) is adjusted (decreasing) as time progressed.
• NoiseSigma parameter seems to be used to set the max. variance of each Gaussian component.
• bgfg_segm ( Co-occurrence and Mixture of Gaussians)
• cvCreateGaussianBGModel() is basically using BackgroundSubtractorMOG as model. Detected object refinement left out. segment_objects did processing at the application level.
• FGDModel is the color co-occurrence model.
• Needs double the memory size of MoG. CPU intensive.
• changeDetection() actually finds the best binary threshold on each color component by choosing one that would give the maximum variance. That's probably based on the reasoning that noise pixel color has small variance. Paul Rosin uses the term RelativeVariance in Thresholding for Change Detection . Here the code compare Standard Deviation. Wonder if simply comparing variance is enough, saving the final 'square root' operation.

Tested with more videos

• Golden Gate Bridge: FGDModel shows no ship moving at all. The slowness is making the ship part of the moving background.
• Sea side beach: FGDModel is able to absorb the cloud and wave (slow) movements and into background. MoG can't.
• Single Moving Tree: FGDModel unable to put the center portion of the moving branches  into background. 
• Relaxing Aquarium: 
• Default parameters of FGDModel takes a long time to adapt to the aquarium after fading-in 
• Slow moving fishes quickly becomes the background of FGDModel.
• Increase 'alpha1' parameter of FGDModel makes the slow fishes appear as foreground. But at the same time the oscillating lights on coral turns up as foreground also.
• Codebook seems to be able isolate the big fishes as foreground, eliminating the fluctuating lights on coral. At the same time the smaller fishes are filtered out. Making it looks like the isolation is due to keeping out small foreground objects all together.

Other readings along the way:

• This article from Intel gives an overview of video surveillance system: fg/bg detection and blob tracking.
• Learning patterns of activity using real-time tracking (Stauffer, Grimson)
• Foreground Object Detection in Changing Background Based on Color Co-Occurrences Statistics ( Li, Huang, Qi )
• Thresholding for Change Detection (Rosin)

Other others:

• There is a flag to indicate use of Least Median of Squares (LMedS) methods in findHomography(). Wonder if that could be refactored for thresholding use.