Tuesday, November 22, 2011

Show Running Video Frame Number

Spent some time figuring out which existing (free of charge) media player could display current frame-number or resolutions up to milliseconds. I focused my search on Windows platform.
  • VLC: Unable to find a way to step frame-by-frame
  • VirtualDub: Yes - it is on by default. Noticed the limited types of media container support. It works with AVI, mpg, but not mp4.
  • Media Player Classic: Yes - but I am only able to show it as OSD, not the status / information area.
    1. Open the video file
    2. Configure the OSD option to display 'frame number' from Media Player Classic menu: Play -> Filters -> ffdshow Video Decoder -> Properties
    3. Turn on OSD option from ffdshow (available from windows taskbar)

Friday, August 26, 2011

Convert RAW YUV File to AVI with AVISynth and VirtualDubMod


Here is a straightforward way of placing a raw video file in YUV420 format into an AVI container.
  • Download and Install AviSynth 2.5.8
  • Download and Place the RawSource AviSynth Plug-in to AviSynth plugins directory.
  • Download and Install VirtualModDub 1.5.10.3
  • Now prepare a simple AviSynth script file called importYUV.avs with the following lines:
file_1 = rawsource("overbridge.yuv", 1920, 1088, "I420")
return file_1
  • Run VirtualModDub
  • Open the importYUV.avs as Video File.
    Now the first frame of video should appear on VirtualModDub window.
  • Choose File->SaveAs to save the imported YUV file as AVI
    • Choose 'Direct stream copy' as "Video mode".
    • Press 'Save' Button.
  • Ta-da! 
References

Tuesday, August 2, 2011

NPP and CUDA

Some highlights of my primitive understanding on NPP and CUDA. The description below does not cover the graphical aspects of CUDA. There are topics on textures, getting resources from other 3D APIs and others.

General notions
Host - CPU, Device - GPU
Thread, Block, Grid, Core
Kernel - a single function to operate on data-array.

NPP - NVIDIA Performance Primitives (NPP_Library.pdf)
It is a set of C primitives that operates an arrays of data. Typical program flow: allocate memory on device, copying the input array to device, call the NPP functions, and copy the result array from device memory.
There are 2 sets of API. First set operates on 1D array - Signals. Another set operates on 2D array - Image.

  • Signal functions (nppsXXX) : arithmetic, set, shift, logical, reduction.
  • Image functions (nppiXXX) : arithmetic, set/copy, stats (min, max, mean,...), histogram, transforms (warp, affine, perspective, domain), color-space conversion, filtering, etc.
NVCC - NVIDIA compiler
The detail compile-flow is sophisticated. NVCC separates the input source (.cu) to run on host and device. It delegates host code to compiler that is responsible for the host application. Device portion of the code will be compiled to intermediate code (.ptx) or architecture-specific code (.cubin), based on the compiler options. Either way, the compiled device code will be embedded in the application binary. At application launch time, the PTX code will be compiled to arch-specific image, and download to device memory. PTX is the output for 'virtual architecture'. It is an abstract version of a device that is characterized with its compute-capability index (1.0, 1.1, .., 2.0,...). The NVIDIA Runtime library will find out what the actual device hardware at execution time and compile the PTX code accordingly.

CUDA - C extensions, Parallelizing Framework ( CUDA_C_Programming_Guide.pdf )
It has nothing to do with NPP (?). It primarily lets host applications to perform parallel data processing using GPU cores (SIMT). It defines a set of C extensions (Appendix B) so that programmer could define how code and data are placed and executed on the device. The framework supplies a set of Runtime API and Driver API. Device API is a lot like a Runtime API. It allows finer control in some cases, e.g. pushing / popping contexts.
Device contexts is similar to CPU processes.
Driver API - cuXXXX()
Typical programming flow: Initialize device - Create contexts - Load module ( PTX or arch-specific-binary) - Choose Kernel (function) from current context - Execute it.
Each host-thread keep a stack of contexts. The top-of-stack is 'current' context. Creating a context for a device automatically push it on top of the stack. A context could be popped from stack. It remains valid. Any threads could pick it up and run it. Omitted from the simplified flow above: calls to copy data from and to device around the kernel execution.
Runtime API - cudaXXXX()
Concept - Single Function Many Data
A Function is called Kernel. It is defined by prefixing __global__ to a C function in CUDA source (.cu). These functions could only be called by functions defined in CUDA source. Typically a Kernel calculates one element of output array.
Each core runs a block of threads by time-slicing(?). A GPU has N cores. GeForce 310M has 16.
The work load of processing an array is spread across available cores by grouping threads into blocks. NVIDIA runtime decide scheduling of these blocks into available cores. CUDA 4 supports up to 3 array dimensions (x, y, z).
Program Flow
Typical program flow is similar to aforementioned - Allocate host and/or device memory - Copy/Map data to device - Launch the kernel with the (in, out) array data locations, and number of blocks and block-size - Copy back the result data to host.
Error handling needs special care to get because of the asynchronous nature. 2 types of error checking: At-Entry (parameter checking) and At-Finish (Kernel function returns)
Syntax
__global__ defines the function as Kernel.
__device__, __const__ defines the variable on device global and constant area.
__shared__ defines the variable to be placed in thread-block memory.
Only supports a subset of C++. See Appendix B of CUDA_C_Programming_Guide.pdf.
A pair of triple-arrow-operator <<<, >>> specifies the data, thread-block-info and launch the kernel. It is ASYNCHRONOUS!
Other aspects
  • Device Memory Hierarchy - Global, Block(shared), Thread
  • Efficient use of memory - Copying from global to block is expensive. Make use of __shared__.
  • Concurrency - Streams ( a sequence of asynchronous commands ). Data copy could be made asynchronous with 'async' variant of data copy functions. Concurrent-data-transfer and Concurrent-kernel-exec depends on GPU capability.
  • Synchronization - Events, Explicit CUDA API, Implicit (device data copy to host, and others).


Monday, August 1, 2011

OpenCV 2.3.0 GPU speed-up with CUDA 4

Now it's time to build OpenCV 2.3.0 with GPU enabled.

Configuration and Build - http://opencv.willowgarage.com/wiki/OpenCV_GPU
Follow the steps described in the OPENCV_GPU page for Visual Studio 64-bit build.
module-gpu build error: Configuration(null)
Solution - missing vcvars64.bat in Windows SDK amd64 directory. Create that by following the simple instructions here http://www.w7forums.com/vcvarsall-bat-no-64bit-support-vcvars64-bat-missing-t6606.html
Taken by surprised at first because I am able to build 64-bit OpenCV. I suspect it has to do with nvidia compiler (nvcc). It probably open a windows shell to do compilation. And that would not have the 64-bit environment set up without this vcvars64.bat.

Test GPU build by running module-gpu-test suite from VS 2010 Express
See "Implementing tests" section of http://opencv.willowgarage.com/wiki/CodingStyleGuide
Setting up Test Data
Test-Data is required by the gpu-test-suite (and others too). Download a snapshot of the opencv-extra package that is tagged for OpenCV 2.3.0 release from WillowGarage. There is a "Download Zip" link in the source browsing page that makes it convenient.
Set the environment variable OPENCV_TEST_DATA_PATH to point to the testdata directory.
Run the project module-gpu-test
Resulted in 3 types of failures
  1. My NVidia hardware that has compute-capability of 1.2. 1 case requires 1.3
  2. Crash in meanShift and meanShiftProc. The stack trace shows that it dies at the point where GpuMat variable is being released.
  3. Assertion error in NVidia.TestHaarCascadeAppl. (Didn't investigate further).
The other tests run OK.

Learned to use the gtest_ command-line argument - see code comments above ParseGoogleTestFlagsOnlyImpl()
  • gtest_list_tests : shows the tests selected to run and quit
  • gtest_filter= : select the tests to run / or not to run by matching a specified pattern against test name. Pattern for negative matching begins with minus sign.
  • gtest_output=xml[: directory name / file-name ] : output a summary of tests results in XML. Details see ts_gtest.cpp (search for GTEST_DEFINE_string_)
OpenCV GPU module
The library implements accelerated versions of other areas of OpenCV  - image processing, image filtering, matrix calculations, features-2D and, object detection, camera calibration. The API and data-structures are defined in nested namespace cv::gpu::. The accelerations makes use of both NPP API and CUDA parallelization.

Run a few OpenCV GPU samples that could readily compared with non-GPU ones
  • surf_keypoint_matcher vs matcher_simple: speed up from 46 secs to 6 secs with the graffiti image from VGG set.
  • mofology vs morphology2 : not very obvious in my quick test. still noticeable when changing the element shape at a Open/Close set at 17-iterations.
  • hog_gpu vs peopledetect : speed up from 67 to 17 secs with my 5M-pixel test image.
  • cascadeclassifier_nvidia_api vs cascadeclassifier(GPU)vs facedetect (no-nested-cascade) : overall (secs): 5.1 / 4.8 / 4.5; detection-only(secs): 1 secs / 1 / 3.1 








Saturday, July 30, 2011

CUDA 4 installation

OpenCV requires CUDA 4 Toolkit. Moreover, CUDA 4 supports VS 2010 while CUDA 3.2 only supports up to VS 2008.

Installation - README_SDK_Release_Notes.txt
Requires NVidia driver of version 270+. Upgraded such for my GeForce 310M.
Installed 64-bit CUDA Toolkit 4.0
Installed GPU Computing SDK.
Also downloaded BuildCustomization FIX just in case.

Verify the installation - CUDA_C_Getting_Started_Windows.pdf
Simply follow the Getting Started guide.
Some hiccups to resolve -
  1. Error building 64-bit 'cutil' - Change the Toolset configuration to Windows SDK 7.1 in order to get $(WindowSdkDir) macro pointing to 64-bit instead of 32 bit. http://stackoverflow.com/questions/3599079/windowssdkdir-is-not-set-correctly-in-visual-studio-2010
  2. Error building shrUtils - the source and include files are misplaced. Copied required source as seen in VC++ Project file from "C/Common/src". Add "C/Common/inc" to shrUtils "Additional Include Directories" to pick up the misplaced headers. http://forums.nvidia.com/index.php?showtopic=197097
Now able to run the build and run the tests suggested in the Getting Started guide.

Applied the BuildCustomization Fix
Not essential but definitely come across such problem in later experiments. So it's still a fix needed for this toolkit. Description of the problem is in the README file that comes with the patch. Quite straightforward, actually.

Friday, July 29, 2011

Install OpenCV 2.3.0 for CUDA

Decided to try out what OpenCV + CUDA is like. Prefer to start with 2.3.0 - quite a few fixes since 2.2.0 in this area.

Simple Build and Run of OpenCV 2.3.0 Release
Download OpenCV-2.3.0-win-src package.
Typical configuration with CMakefile for VS2010 Express - with C samples.
Build 32-bit - run video-starter to check video-capture is working - YES.
Build 64-bit - run video-starter to see video-capture is working - YES.

Build API documentation to understand GPU module
Download and Installed MikTex 2.9 Portable version to C:\Program Files (x86).
Configure CMake: BUILD_DOCS=yes and set MIKTEX_BINARY_PATH = < miktex/bin directory >
Needs Sphinx python module (sphinux-build.exe) to satisfy CMake configuration of Build Documentation
  • Install Python 2.6.5 (Win32)
  • Install setuptools
  • Download sphinx python egg
  • Run easy_install (from setuptools) on the egg.
  • Run CMake config again, specify exact path to the sphinx. This entry now appears now that Python installation is detected.
Open OpenCV.sln with VS2010 Express - ALL_BUILDS configuration currently excludes 'docs' and 'html_docs' project.
Compile 'docs' first - error at the end(?) saying the pdflatex.exe: Access is denied. Seems like pdflatex is trying to write some file within the Miktex program directory. Next time I should move the MikTex Portable directory to C:\ProgramData instead.
Build 'html_docs' - finished OK; HTML API docs now appears in the build directory.

Also noticed:
a WITH_OPENNI option in CMake - and there is a kinect_sample.

Thursday, July 28, 2011

Speed up with Intel Integrated Performance Primitives (IPP)

Current OpenCV support
OpenCV 2.3 : IPP 7.0
OpenCV 2.2 : IPP 5 - 6.1
The directory structure of IPP 7.0 and IPP 6.1 is different. And the OpenCV 2.2 CMakefile only checks for IPP library versions up to 6.1.

Licensing
It has both commercial and non-commercial license. A single-user commercial license costs $199 + $80 for annual-renewal. Non-commercial version only supports Linux.

Installation
Tried the 30-day evaluation of non-commercial version. Surprised it requires 2GB disk space for installation. The package itself is about 237MB.

What is IPP?
Relationship with OpenCV according to Intel - outdated with respect to current OpenCV code.
http://software.intel.com/en-us/articles/intel-integrated-performance-primitives-intel-ipp-open-source-computer-vision-library-opencv-faq
IPP uses OpenMP for parallelization
http://software.intel.com/en-us/articles/intel-integrated-performance-primitives-intel-ipp-intel-ipp-threaded-functions/
Did not spend time to go into details. My impression is it provides a lot of OpenCV routines in image-processing, camera-calibration, optical flow. And quite of few of them have a 1-to-1 mapping to the corresponding function in IPP. Surprised to see so few area actually uses IPP now. Based on what I read from the discussion forum, IPP speeded up OpenCV1.x a lot, before having SSE acceleration.
SSE is instruction-set level hardware acceleration. IPP is a library that implements algorithms that takes advantage of SSE.

How is it used
Did a quick search of 'HAVE_IPP' from OpenCV 2.2 source code. These are areas currently IPP appears to be relevant: dxt,cpp (e.g. DFT), haar.cpp, hog.cpp, LK-optical-flow, HS-optical-flow.
Something I noticed is how the IPP library is loaded at run time. There are a few linking methods to choose from at build time. I think what OpenCV 2.2 is using is what is called Static Linking with Dispatching - ippStaticInit(). At build time OpenCV linked in a static library which provides a 'jump-table' to the actual dynamic-library at run-time. The IPP routines are optimized differently across processors based on their capabilities. ippStaticInit() chooses the suitable library to load based on the processor it's running on. See the IPP User Guide for details.

CMakeFile Configuration
Downloaded and installed IPP 6.1 Build 6 to Linux.
Point IPP_PATH CMake variable to the IPP lib directory, not IPP root directory.

Comparison
dft sample - the time took to complete dft() function reduced from 7XX ms to 3XX for an 5M-pixel image file.
facedetect sample - 5% speed up of face-detection function using haarcascade_frontalface_alt.xml. Some classifier will not trigger IPP enhanced code. See haar.cpp for the conditions required.