PixelNN – Example-Based Image Synthesis

Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation".

You fundamentally can't get back information that has been destroyed/or never captured in the first place.

What you can do is fill in the gaps/information with plausible values.

I don't know whether this sounds like I'm splitting hairs, but it's really important that the general public not think we're extracting information in these procedures, we're interpolating or projecting information that is not there.

Very useful for artificially generating skins for each shoe on a shoe rack in a computer game or simulation, potentially disastrous if the general public starts to think it's applicable to security camera footage or admissible as evidence...

Out of sheer curiosity I had a go at manually enhancing the Roundhay Garden Scene by dramatically enlarging the frames, stacking them, aligning them, erasing the most blurred ones and the obvious artifacts.

It went from this:

https://media.giphy.com/media/pUf3YfamV7BV6/giphy.gif

To this:

http://img.go-here.nl/Roundhay_Garden_Scene.gif

The funniest part was that the resolution really goes up if you make 1 px into 40 and align the frames accurately (then adjust opacity to the level of blur)

The crime television thing would be possible if you have enough frames of the gangster.

Approaches like these are hallucinating the high resolution images though--not something that we'd ever want being used for police work. That said, I wonder if it would perform better than eyewitness testimony...

This image from the article shows that the original image and the fantasy image are not alike at all. The faces look to have different ages. The computer even fantasized a beauty mark.

http://www.cs.cmu.edu/~aayushb/pixelNN/freq_analysis.png

The computer is fantasizing.

The low resolution to high resolution image synthesis reminds me of the unblur tool that Adobe demoed during Adobe MAX in 2011. Here is the relevant clip if you're interested https://www.youtube.com/watch?v=xxjiQoTp864

You'll get something that looks plausible for sure, maybe not what was originally there though. In the future, someone will be falsely convicted of a crime because a DNN enhance decided to put their picture in some fuzzy context.

It can give possible matches, i don't think it would be admissible in court. they could still trick a confession out of someone using that image.

It's still impossible. These algorithms find in gaps with their biases, not reality. If information is not there it is not there.

Yes!

Although what we don't have is any certainty that the enhanced face actually looks like the killer.

Except that it can put trees somewhere where there were no trees but something similar to them. Or it can put face of a more popular actor instead of an actual less popular one because it was more often present in the training dataset. No, thanks.

Plug in the script and some artist's impressions of the sets and generate the whole movie on the fly.

That's what video compression does now.

You never saw edges2cats I take it? https://affinelayer.com/pixsrv/

> I don't understand how the edges-to-faces can possibly work. The inputs seem to be black & white, and yet the output pictures have light skin tones.

The step you're missing is that an edge detector is run on the entire database of training images to produce a database of edge images. The input edge image is run against that corpus of edge images in order to find which edge images match, then sample the corresponding original color images and synthesize a new color image.

I had the same thought. Maybe it's not that there were only white people in the dataset, but it's actually taking the shape of the face into account, and it most closely matches those with white skin tones. I suggest this by looking at the cat one: it has the stripes coming off the eyes, so suggests one of the grey striped breeds rather than, e.g. all black or calico. It's probably more than pixel-by-pixel NN interpolation, but also taking into account some of the actual structure of the edges.

Color comes from the initial neural network step. Since skin color is relatively predictable from facial features (ex: nose width), it should be able to do reasonably well.

That's perceptual hashing. Check out https://www.phash.org/

This is the current approach for large sale image retrieval. By using some model to extract features and then performing distance calculations. This is usually done with hashing once speed and the size of the dataset become large.

I am developing a Visual Search engine with pluggable indexing models.

https://www.deepvideoanalytics.com

Yes, this is a fairly similar concept: https://magenta.tensorflow.org/nsynth

This is actually training a neural network on the Markov model, so it's very similar to core ideas behind the OP's paper. The core idea is to model the probability of a bit of sound by breaking it into the last note and everything that comes before the last note ("P(audio)=P(audio∣note)P(note)"). If you sample a bunch of audio and factor it that way for any given point in time, and accumulate that data somewhere, you can then sample the accumulated data randomly to generate new music.

There are other audio NN synthesis methods as well, pretty sure I've even seen one posted to ShowHN before.

There kind of already is audio equivalent: MIDI. It supplies low resolution timing and pitch information and it's up to synthesizer to produce audio output matching those data.

What would the lo-res starting point be? Low sample-rate, bit depth, ...?