Blob Tracking - Video Surveillance Demo

A blob tracking system is included in OpenCV
Code: OpenCV/modules/legacy/
Doc: OpenCV/docs/vidsurv/

The blob-tracking code consists of a pipeline of detecting,  tracking and analyzing foreground objects. It should be a video surveillance system demo by the name of its folder. It not only detect and track blobs, it tries to pick out unusual movements with the analyzer.

The main loop of processing the input video frames is CvBlobTrackerAuto1::Process(). And the order of the processing order differs a little from the Blob_Tracking_Modules.doc. For example, the BlobDetector is actually run after BlobTracker and Post-Processing.

Each stage has a few methods to choose, for example, I could select between FGD and MOG for the FG/BG separation stage.

Some notes I skim through some of the code of the following stages:

Blob-Detectors  (BD_CC, BD_Simple)

• The file header suggests reading literature by Andrew Senior et al (see Resources section).
• The purpose of this module is to identify blobs that are 'worthy' of tracking. It does so by first filtering out noise from the foreground mask. The main criteria is that the CCs has to move in a reasonable 'speed' (uniform motion). It determines so by keeping a list of candidates connected-components for the last 5 frames. That probably means that the frame-to-frame change of blob location does not exceed a certain amount in order to qualify.
• Well I cannot tell the difference between the BD_CC and BD_Simple method by looking at the code.

Blob-Trackers (CCMSPF, CC, MS, MSFG, Particle-Filter)

• Some literature listed at blobtrackingccwithcr.cpp regarding Particle Filter and tracking temporarily obstructed objects. 
• Some code are there for Multiple Hypothesis Tracking (pBlobHyp) - but execution breakpoints were not triggered during my testing.
• The CC trackers uses Contour and Shapes (Moments) to represent blobs while the MS-based uses color intensity histograms. 

Both Connected-Component trackers use Kalman Filter to predict the next blob location. The ProcessBlob() function updates the blob model by the weighted sum of newly captured value and the prediction. In the case of blobs collision, only the predicted value would be used.

Collision Checking

• CC Method: Detect collision by exhaustively examining every Blob (position + size) 
• CCMSPF Method: Go further by 'resolving' collision with a MeanShift-Particle-Filter.

All MeanShift trackers below are an instance of CvBlobTrackerList. It holds a list of the corresponding CvBlobTrackerOneMSXXX instances. One tracker for each blob.

• MS Method: Simple mean-shift tracker for every blob. 
• MS with FG weights: Foreground pixels are weighted during calculations. A higher value makes the blob accelerates the movements and resizes itself in the model. 
• MS with Particle Filter: 200 particles is allocated for each Tracker (Blob). Each particle represent the same blob moving and resizing a little differently from others. It position and size delta is generated within some preset variances plus some random value. At the each frame, the particles got randomized with the new values with those parameters. And a weighted sum yields the new prediction (position, size). And then the particles are shuffled and the weights are reset to 1. Each particle is associated with a weight. The weights are updated every frame. They are functions of the Bhattacharyya Coefficients calculated between the current Model Histogram and the Candidate Histogram. The Model Histogram is updated every frame from the blob position and size. The Candidate Histogram is the histogram calculated with the hypothesis particle. Where is the mean-shift?!

Condenation Algorithm ((Conditional Density Propagation) - to represent non-Gaussian distribution. When an object is obstructed, there are multiple possibilities of what could happen. And that is not Gaussian. My understanding is the Particle filter is able to represent multi-modal distribution. The distribution is represented by groups of particles. The density of each group represent the probability density of one of the range along the x-axis.

Post-Processing (Kalman Filter)
Results from Tracking stage will be adjusted by Kalman Filter. That is the Blob Position and Size will be updated.

Track Generator

• Record the track (position and size) of each blob to a user-specified file. 
• The values of both information are represented as a fraction of the video frame size. 

Blob-Track-Analyzers ( Histogram analysis of 2D, 4D, 5D, SS feature-vectors, TrackDist, IOR)
A 'status value is maintained on all active blobs:  Normal or Abnormal.

TrackDist

• Tracks of Past Blobs (no longer a foreground in recent frames) are added to track database. They will be used as templates to compare with the active blobs tracks. 
• Find the closest match from the templates for each active blob in terms of their similarity in position, velocity. 
• The state is Normal if it could find a reasonably close match.

Histogram P, PV, PVS

• Each active blob has a histogram representing its track. There are 3 types of dimensions: 1) position, ) position and velocity, 3) position, velocity and state-change. As far as I could understand, the state-change represents the number of successive frames during which the blob moves very slowly. The 'slowness' is making it almost stationary between frames. 
• A sparse-matrix is used to store the histogram of these continuous vector values. 
• Nearby histogram bins are smoothed at which every new vector is collected. 
• All histograms are updated at every frame. 
• Past Blobs will have its histogram merged with a global histogram. Similarly, it will be used to decide whether a particular active blob track is 'normal'.

Histogram SS
Similar to P-V-S histogram except that the vector consists only of starting-position and stop-position. A blob would be seen as stopped as soon as the state-change counter reached 5.

Demo Code (blobtrack_sample.cpp):

• The demo code put everything together. In its simplest form, user supplies a input video file, it would display the 'Tracking' window - marking the moving blobs on the video with a location-size-circle, BlobID and Analyzer status (Abnormal in red, Normal in Green). 
• The tracking video could be saved to a video file if provided a file name. 
• The foreground masks could be shown at a separate window and save to a video file of user's choice. 
• If a Track-Generator output file is specified, the precise blob locations at every frame together with its frame-of-entrance will be would be recorded to that file. 
• And there is a general log file showing the user parameters, same as those appear on the command console.
• Although haven't tried it, user should be able to pass method-specific arguments. For example, in addition to choosing Meanshift method for tracking, user is also able to pass specific parameters. See function set_params().

Results from Road-Side Camera Video)

Command-Line Arguments: bd=BD_Simple fg=FG_1 bt=MSPF fgavi=fgSeparated.mp4 btavi=blobtracked.mp4 bta_data=trackAnalysis.log track=track.log log=blobtracksample.log

In general I am not too satisfied with the results in terms of tracking. I don't know whether this is expected with video I have. For example, the blobID for the same car could change as it goes farther from the camera, and vice-versa. The analyzer result is often abnormal for some reason even if the cars are simply going along the road.

Resources
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/ISARD1/condensation.html
cvBlobsLib: http://opencv.willowgarage.com/wiki/cvBlobsLib

Reading
Learning OpenCV, O'Reilly Press.