▲ 108 points
back
59 comments
The author of the exceptional neural net (DeOldify) behind this is Jason Antic. https://twitter.com/citnaj
The funny thing is he has been kind of attacked on Twitter recently for bizarre reasons ... people are pointing out how colorization detracts from the source material, or ruins history, etc.
What this really is, is a hacker making the world's best neural net in the world at a certain task. (It appears to have also become his day job now.)
Let's try to appreciate what an achievement it really is. Pretty much all of the automatic colorization you may have seen online in the last year is some version of DeOldify.
After watching both versions, my brain processed a different story between the two. In the colorized one, I see a bunch of people having a random good time, then some passerby gets caught in the middle. And although he tries to escape, he doesn't seem too upset. This results in a feeling of mild entertainment. Kind of like the same feeling from watching a video of household pet antics, or similar.
In the original, I see a bunch of Vaudeville "clowns" doing their antics, and a "straight-man" appears on stage, and tries desperately to escape the situation. The result is a feeling of the film attempting to induce mild amusement. Somewhat like watching a film where someone slips on a banana peel, or sits on a breakaway chair (slapstick comedy).
I believe that this is due to all the exposure I've had of the Vaudeville material has been based on grainy jittery monochrome film, so I interpret the material based on the typical Vaudeville setup.
This is true even with still photography.
I still remember an example from a series of photography books that my parents purchased for me when I was teenager that went into this.
Two scenes from an inner-city neighborhood of children playing and adults standing around. The black and white version emphasized the general grittiness. (Which by that point in the 1970s we were proabably at least somewhat conditioned to often associate B&W with.)
By contrast, the color version had lots of bright colorxs and really emphasized the children happily playing.
The video was a staged (I believe) silent film[0], so it seems like the impression you got from the original black and white version was more "accurate".
[0]https://en.wikipedia.org/wiki/Bataille_de_boules_de_neige
How much of that difference was due to the different soundtrack? I noticed the difference as well and attributed most of it to the slow and sappy music of the upscaled version versus the fast and irreverent music of the original.
Also a rather significant achievement: people are actually watching these old videos. Seems like it stands a chance of being the widest distribution of such content ever, by a large margin. When was the last time most of us consumed century-old media of any kind?
Though I understand the historian side of things... every time I've seen major colorizing or cleanup efforts, I've seen far more people looking at them and talking about the history around that time than I'd normally see in, like, a year or more. Feels like a massive net-win to me.
> I understand the historian side of things
You mean the contrarian side of things? There isn't a big movement of historians against this.
Classical music, Shakespeare, da Vinci, the Bible?
The difference with a colorizing, cleanup or restauration effort of a material object is that in that case the source material is not destroyed.
In that regard, the only downside for historians is that it might become more difficult to find the source material if upgraded versions of it proliferate, but this is not like rewriting or erasing history like some have claimed.
To me it seems like there has been a much larger issue of photographs, movies etc. being digitalised at some point years ago using some standard of the time, after which the originals were discarded because "there, now it's saved and the old ones are an inferior media anyway".
Then you're left 20 years later with a horribly jpeg'd mess in 800x600 px.
I don't have a source on this, but it's my impression a lot of libraries did this in the 90's or 2000's (if someone knows specific cases i'd love to hear about them, I hope I'm wrong)
There was a lively post on reddit discussing the ethics of colorizing historic photos a few days ago:
https://www.reddit.com/r/OutOfTheLoop/comments/j523qi/whats_...
I found this point interesting:
"The thing about old photos is that everything about the image is part of the information we can learn from it—not just the picture itself and what it's a picture of, but how it was composed, what its medium was, the balance of lights and darks... It's all part of the document. And when you start to alter it, that's creating a new document reflecting a conjunction between the past and the present. "
Surely this is only a problem if, for whatever reason, a significant number of people are led to believe that the altered footage is an accurate historical representation when that is not the case. People create media that is not an accurate historical representation all the time: fictional movies, cartoons, etc. and presumably historians aren't upset about that.
> presumably historians aren't upset about that
Few historians will rail against an inaccurate portrayal of historic events in fiction. For one, many enjoyable books or movies would be rather dull if authors didn't have the liberty to beef up the story.
However, historians will pipe up when fiction is taken for a historical fact. That's what historical research is all about: uncovering the facts and testing how those facts support different theses, narratives and perspectives.
This debate though, isn't about misrepresenting history through fictionalized stories. This is about enhancing the quality of the footage itself and how that impacts our response and understanding of the past. Which is something else.
The issue here is the intent with which such enhanced footage gets purposed and repurposed later on. Today, it's sights and sounds of cities, or a snowball fight. Tommorow, someone might apply those same techniques to reels of film which are culturally, socially and politically sensitive in order to elicit a specific, strong emotional response with their target audience.
If done subtly, an unsuspecting audience might never even realize that they are watching enhanced or re-touched footage, instead assuming or accepting that the footage was made during that era.
That's what is pointed out here: by enhancing the footage - removing the artefacts that show it's true age - you also remove barriers that create a gap between you and the past. For better or worst, that is.
The issue is not about the “accuracy” of a filmed historical subject but rather of overwriting a new set of authorial and editorial choices over the original. If you’re trying to understand how the author thought of the world in order to understand a historical mindset, you look at the choices they make as reflective of their world view and values.
Worth pointing out that the original media itself is compromised as an accurate historical representation.
I approach this from the viewpoint that we have a huge amount of historical media and writings that most people are never exposed to, and altering this historical media in a way that will get it more exposure increases peoples' understanding of the past.
My 11 year old had little interest in World War II until he saw the series “World War II in Color” on Netflix. He’s watched every episode and now reads books on the subject.
Exactly. What use is a well-preserved grainy, flickery video that no one watches? Even more, how do we understand history if we only see it through a blurry, scratchy lens? It might change our perception of the people who lived 100 years ago if we can only imagine them in a limited color scheme.
I agree that the originals shouldn't be lost but there's no reason not to do restoration if it is done carefully or tastefully.
it's not like the original is being deleted though.
That discussion happened here as well. https://news.ycombinator.com/item?id=24649770
Funny thing, the top ‘trending page’ on the OP site is: “Stop upscaling and colorizing photos and videos, historians say”.
Synthesizing trumped up controversy is a great way to get coverage.
IDK. Personally I think it's silly to romanticise and wax lyrical about what are - when it comes down to it - errors in the data; artefacts of the crudeness of the technology that was available when the material was created. Do we routinely add errors of this type to modern footage? No. Why not? Because they detract from the material.
The material is already hugely compromised as a historical document of the reality on the day the events were captured. Let's not put it on a pedestal. For example, if we remove the effect of hand-cranking on film to make it more watchable, are we destroying or enhancing? I would argue the latter. The hand-cranking adds little of value.
I saw this on Reddit the other day and have read some of the worries from historians about changing history.
I view this as one half of the equation, if people can relate to the past more (like WW2 in HD, They Shall Not Grow Old) etc, then suddenly they can connect and understand that human problems have always been around and we aren't so different.
As long as we can balance that effectively with the historical preservation part I'm all for using things such as ML upscaling/recoloring etc in order to make the past much more accessible.
I had the opportunity to see the Apollo 11 documentary on the IMAX last year when it came out, it was one of the most incredible movie experiences of my life and helped me to "get it" in regards to how the people acted, the public's reaction, enthusiasm etc.
That colorized well, or seemed to, since the original had little color. It had to color "building", "snow", and "skin". That's about it.
Here, though, is a similar technique applied to the Wuppertal monorail.[1] The colors are all wrong. The supports was really lime green, the overhead track was brown, and the carriages were a glossy red. Since the training set lacks data for 19th century supported monorails, it's just guessing.
That's a problem with the colorization for all of these. It's also just not very good. Faces are all a uniform color, sky is a grey-ish blue-ish color, ground is greenish-brownish. It adds color, but, as you point out, it's just guessing, and it's not guessing very well for the most part.
Looking at both, I like the BW version better. The updated version makes it look like any other 1 minute clip of a snow fight using a bad phone camera. It's an interesting exercise on how to use tech but ultimately it takes away from the original clip rather than adding to it.
> makes it look like any other 1 minute clip of a snow fight using a bad phone camera
But isn't that exactly the point?
Instead of making the past "exotic" and different and remote... it makes it seem just like us now.
Because let's face it: people were the same back then. But because of how serious and black-and-white photos usually are, we're usually led to believe the past was far more alien than it was.
Would be nice to be able to pick and choose. In the original the exposure is all over the place, and stabilizing it makes it much easier to watch. Smoothing out the camera shake makes a big difference. I can see the benefit of de-scratching. The colorization doesn't contribute much.
This is amazing. The uncanny valley is almost entirely gone; the people actually seem real.
It looks even more realistic if you set playback speed to 1.25x.
You're right, huh. There's nothing "objective" here to match the frame rate here to (the cameras were hand-cranked AFAIK)... it does feel like the speed is a little slower than reality, though 1.25x feels a tad too fast.
I do wonder why they didn't bother to fix that a little?
Not mentioned but there is also stabilization, which helps immensely.
> The colorized version was first posted to Twitter by Joaquim Campa, who used the AI-powered software DeOldify to upscale the footage to 1080p
So they started from a low-res scan, because film is not SD, you can do a 4K scan from film too. Of course you wouldn’t necessarily have 4K of useful information, it might just be grain. But it’s interesting to think that many of these videos probably start with subpar digitizations of the source material, and wonder what it would look like when based on a proper high resolution archival scan
With a lot of these techniques, you can't just start with better source material. The frames have to be resized before going through the neural net, so that they fit its expectations.
So if you want to start from a 4k source, you have to train your network on 4k data, and that becomes a ludicrously expensive exercise quite quickly.
Once the artifacts of rudimentary film are no longer there, I was better able to focus on other details.
Like, they're just doing this in the middle of the road! Very few automobiles to spoil the fun.
The colorization is laughable, but the upscaling is beautiful.
The original version at the link is not only black&white and not upscaled in resolution, it also has jerky mis-aligned frames. I'd think there would be a middle ground here, in which that sort of artifact is removed, without adding things like colour, which can only be added by making assumptions that have no basis in the actual images captured.
The "colorized" version is very brown. Is that a limitation of the AI or is that really what it probably looked like?
At least some attempts at AIs colorizing photos, that were previously posted on Reddit or HN, tend to break into sepia on many things—and for some reason, also blue on clothes. Even when it clearly looked unnatural. So it seems to be an artifact of current approaches in general, and not of this particular case. (However it might be that this post and those ones were made by the same people.)
Personally, from my completely-uneducated viewpoint, I wonder if this shows the current boundary of what's possible, with the AI not having the knowledge of what it sees. Presumably you need to first teach the AI about all of the things in the old photos/videos, and then somehow show what color each of them should be. Particularly, I'm rather skeptical about the latter part because e.g. clothes have plenty of variability in color, and properly you'd need to guess the kind of clothes, the material and the period to take a stab at the color. Like, no blue in the middle ages, except for rich people.
Conversely, I'd imagine that if the AI is trained on ten different examples of pants of various colors but about the same darkness, it can't pick any particularly saturated color because the choice would be mostly random. But ‘drab’ is always safe.
Unclear. These upscaling techs are more "this is what it realistically could have looked like" than "this is what it did look like"
This seems to be an artifact of the training error used by some AIs: in case a simple L2 loss is used, the AI will generate close to an average color (average over an "intrinsic" color distribution), which appears washed out.
I wonder if some enthusiast is going to find these building in modern Lyon and compare the look.
What's the software used for this? (And is it open source?)
DeOldify
The article says 1886, but the caption on the video and Wikipedia both say 1896, which seems much more plausible.
I believe you but out of curiosity why does 1886 seem "much more plausible" than 1896? I don't think I could have picked correctly from those two dates without straight guessing even if it was the last question on who wants to be a millionaire.
Ok, we've added 10 years above. Thanks!