Are you using the rectangle detection from Vision? https://www.youtube.com/watch?v=wt96OomJY9A
Yeah, I am using Vision's rectangle detection.. but it's really touchy. I've got a lot of heuristics in there to throw out bad results and smooth things out.
It doesn't do well if there's something near the puzzle though (like a horizontal rule or text like you might see in a newspaper layout).
Any suggestions on a better way to segment the rectangles? Or any preprocessing tips (I couldn't really find anything that made it detect things more reliably)
https://www.cmgresearch.com/2009/08/14/How-does-it-all-work....
My approach was to adaptive threshold the image and then assume that that the object of interest was the biggest connected region in the image. Then run that though rectangle detection.
Have you tried setting the aspect ratio and minimum width on the Vision rectangle detector? That might filter out any nonsense.
Whatever I do though I can't get it to recognize any rectangles here: http://www.telegraph.co.uk/news/science/science-news/9359579...
For our demo video we ended up photoshopping out the horizontal line right above it and that seemed to work.
I think you'll have to roll your own rectangle detection and use heuristics to make it better.
So in mine I assume that the connected object in the thresholded image with the largest number of pixels must be the puzzle. That makes it a lot easier.
Good luck!