back
23 comments
I would not compare a fancy NN algorithm to Bicubic interpolation, even as an example. Bicubic sucks. If someone really wants to get a decent high-res upscale of an image via "conventional" means (like GIMP) they would probably use something like Lanczos algorithm for upscaling -> Anthropomorphic smoothing for denoise -> unnsharp mask for clarity. And that's a simple pipeline, you can go even fanciers with masks, layers and edge-detection kernels.
They aren't. They compare to techniques "A+" from R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution" http://www.vision.ee.ethz.ch/~timofter/ACCV2014_ID820_SUPPLE...

and "SRCNN" C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution” https://arxiv.org/abs/1501.00092

Their result seems not to score as well in quantitative metrics (signal to noise ratio), but is faster. (And seems to have good qualitative results, in my opinion.)

Indeed, I just skimmed through the paper, and the results don't seem much better than A+ and SRCNN (even worse in some instances).
Yes, they aren't notably better, but they are a lot faster. Maybe useful.

I really wish people wouldn't call these algorithms super-resolution, as they already had a useful meaning.

I think superoscillation is a better description of the phenomena in E&M/optics.
I'm speaking about the pictures on the website OP links to. They even say "As you can see in the example below, the upsampled image looks blurry – one would hesitate to call it enhanced." It's blurry, because they use an algorithm that produces blurry images. Lanczos, for example, would produce a much sharper picture at the expense of some minor artefacts.

They don't show A+ results anywhere in the blog-post at all.

Those looking for quantitative comparison of this against other techniques, it's on page 21 of the paper ( https://arxiv.org/abs/1606.01299 )

Edit: and to see for yourself in more detail there's https://drive.google.com/file/d/0BzCe024Ewz8ab2RKUFVFZGJ4OWc...

https://github.com/nagadomi/waifu2x

Here's a previous project that attempted this as well.

https://arxiv.org/abs/1501.00092

This is obviously something where machine learning is a great tool, knowing about the way images usually look, it's easy to imagine deep learning methods doing a great job at this.

However, an obvious application of this will be to apply these methods to security image footages, in order to improve the recognizability of the people shot. You know like in Hollywood movies where the detective asks the geek "can you improve the details of that picture?" But the general public (and by this I mean cops, lawyers and jury people) will not understand that the details are actually made up. The person in the security footage may be unrecognizable before treatment, but after, it would look as "the most likely person to have been there."

What could go wrong?

Are there any machine-learning approaches to superresolution which take context into account? By that I mean filling in details that can only be guessed by context.

For example, when the algorithm detects that the image is of a field of grass (but individual blades of grass not discernible), then in the superresolution output it paints in these blades.

One of my favorites is Neural Enhance: https://github.com/alexjc/neural-enhance

I was playing around with it and the results are quite interesting. It's pretty much exactly what you describe.

The nice thing is, if you have Docker installed, you can run:

  docker run --rm -v `pwd`:/ne/input -it alexjc/neural-enhance --help
and everything magically works. Compared to other machine learning projects, neural enhance is probably one of the easiest to get started with.
That looks amazing! Thanks.
Some of the deeper - and wider - convolutional neural network-based SR methods do this, implicitly. We have a few papers, like SRGAN (https://arxiv.org/abs/1609.04802) or AffGAN (https://arxiv.org/abs/1610.04490) that generate pretty interesting context-aware textures. Sophisticated hallucination of missing details comes at a cost: As one would expect, in some cases the algorithm guesses incorrectly. See for example Figure 5 of AffGAN paper, column 3: when superresolving an image of a snake, the algorithm mistakenly thought it's a body of water, and produces textures accordingly. Overall we found that most people find the results of GAN-based approaches superior to other ML-based techniques.
This is very clever, and an interesting read, but kind of disappointing results.

They don't look much better than doing say... upsample to twice desired size, apply basic sharpen/structure filter, downsample to desired size.

A little better? Yes. But some (like the horse) look pretty weak

They're never going to look as good as you might be expecting from films. That would require fabrication of information that doesn't really exist.
The best methods for this, using Generative Adversarial Networks, do in fact fabricate new realistic-looking information.

I think it would be great to input a blurry version of a cityscape, and have it add numbers to mailboxes, litter to the streets, cracks to the sidewalk, etc.

Sure, though that is actually possible - generative methods exist.

I just feel it's a lot of (cool) work for very little visual benefit.

It's neat, but the results are disappointing

I wonder how it compares with Waifu2x
Does this mean we can finally get a form of Zoom-and-enhance seen on CSI?
I'm imagining the day when old black and white films and home movies are restored to be almost crystal clear.

Can't be more than 18mo away by the sounds of it.

I was thinking of perfect restorations of old classical music recordings. If we decode the sound of a violin, we could re-synthesize it in studio quality.
From a qualitative glance at the images, I believe I could come close to the same results with some basic filtering, like a gussian blur followed by a quantization to a smaller number of colors, then a little bit of blur.

The image of the horse head statue looks like an "oilify" effect.

Mind you, a pretty good oilify effect.

This filter is clearly useful independently of what it achieves in relation to what it "says on the box".

In creative image processing, I just have to like what it does, whatever that is (and I do).

Is there an executable form / source for this we can play around with?