Picked this up while looking for fitellipse() from OpenCV discussion group:
In my current project I use cvFitEllipse to find ellipses. First, I use cvCanny to detect edges. From this egdePixels I randomly draw 6 Pixels and fit an ellipse. Afterwards, I confirm this ellipse by counting the number of edgePixels lying under this ellipse.
The main problem is to choose the right set of edgePixels to draw from, since you need to draw six Points on the Ellipse to have a Chance to find the right ellipse.
This is approach is calles "Randomized Hough Trafo" and works quite well.
This paper proposes this method?
This article gives an overview of the advances of RHT.