▲ 328 points
back
304 comments
To be honest, I wouldn't be surprised if Apple isn't already doing this and just doesn't say so explicitly. There is a crazy amount of image processing going on behind the scenes in each smartphone photo. Would be interesting to compare a photo taken with the native camera app and photosynthesis to compare. I'm assuming your demo images on your site are the raw and processed images from your app that don't go through Apple's "secret sauce" processing pipeline.
Hijacking this comment because of this line that reminded me of something:
"There is a crazy amount of image processing going on behind the scenes in each smartphone"
I bought the Pixel 10 because it said it had great zooming capabilities. After 5X (that means 10X), it's completely stupid. Basically it takes a very blurred photo and then tries to reconstruct it with AI. The final results are completely awful and really distant from reality.
Any advice in how to improve that?
Yes exactly, the site shows photos all taken from within the app. I have done A/B tests against photos taken in Camera.app and the Photosynthesis composite does significantly beat the detail central area covered by the second lens in Photosynthesis.
Because of the limitations of capturing multiple lenses at once through isVirtualDeviceConstituentPhotoDeliveryEnabled, there is a second capture mode in the app for 'Quality' which takes a simultaneous capture then takes sequential shots from each lens which do benefit from Deep Fusion, then uses those for the stitch (as long as the Deep Fusion images outperform the simultaneous capture results). That's also what I do for our Night mode.
Phones have been doing this for a LONG time, if you had a oneplus 7 pro (released may 2019) and took it into an extremely dark area, like where your eyes can't see anything at all, and take a 'night mode' photo, it asks you to hold it steady for about 4-5 seconds and is clearly taking a series of photos with long exposure time. Then it stacks them and uses data from the sensor of hand movement jitter to deblur.
The result when done outdoors (like, standing in a dark forest at midnight) will be a photo where you can actually see what's around you, and details show up as if it was a brightly full moon lit night.
I use iPhones (paid for by work) as a tool for my agronomy job. I take a large amount of photos throughout the day to help document what I find in the field.
Since at least the iPhone 12, holding the phone still and letting scene sit for a few seconds will result in a sharper photo. I’m not sure if it’s a subtle version focus stacking or a sharpening algorithm, but it’s definitely there.
On my 16e, immediately looking at a photo just taken can result in the image noticeably sharpening after a second or 2, so this may be a post processing step as well.
I think this is just Deep Fusion.
It is part of Apple’s “secret sauce”, yes.
One time I took a photo of a snake in the road with my iphone in low light. When I looked at the photo afterwards the snake was removed by the AI enhancement
Reading title and this ^ comment before loading the page made my brain assume that this is about combining front and rear image sensors, and additionally using eye/facial tracking to infer user intention + satisfaction to help with stabilization/framing/post-processing. I'm glad(?) we aren't there yet.
This should be easy to check. In these photos, the center of the image has higher detail than the edges.
Somewhat related... I made an indie app called TwinCam which combines front-and-back camera images to create a combined selfie. Free. No strings attached. Cheers!
https://apps.apple.com/us/app/twincam/id6448645623 https://play.google.com/store/apps/details?id=org.evergreenl...
Neat work, but the free tier lets you export 3 images a month and the alternative is a subscription? No thanks.
It would have been nicer if the examples wouldn't have been digitally altered to exaggerate the results. For instance, the center of the third photo has been intentionally blurred in the "standard photo" version.
The fact that they have to resort to manipulation to sell their product tells me that they are not trustworthy and that their product is probably underwhelming.
It’s weird, because the center of the image will be more detailed, so how are they dealing with the “grid of pixels” that is on the display. Are they doubling up the wide angle.
“Easy-to-understand framing The brackets over the viewfinder indicate the area covered by the more detailed lens. Frame your subject inside them to capture the best detail.“
Reminds me of the old 3d cameras that took two photos at once through 2 lenses (on film)
https://www.ebay.com/itm/206409326865?chn=ps&_trkparms=ispr%...
I just assumed this was what all phones with multiple rear cameras were doing, is it not? What are the multiple cameras for other than that?
subscription camera app seems like peak comedy to me
If it works, Apple copies it soon enough...
The principle is similar to foveated imaging [1].
Recently, Samsung introduced the ISOCELL platform which allows to select the area which will be recorded at higher resolution compared to other regions.
There can be all sorts of different applications build around such as extracting higher resolution of text regions, etc [2].
Of course, the lens needs to support this resolution but I think that will be default in the mid-term future.
Why do i need a subscription for a camera app?
I am co fused, 50% of photos look better in standard zoom and 50% of them look better woth photosynthesis. This is like an anti campaign for the app hinting "in the end it doesn't really matter"
I cannot see a meaningful difference at all in any of the example photos, so not really seeing what the point of this is.
Not a fan of “Photosynthesis” on the demo. Sliding between the two, “standard” looks clearer and cleaner.
Left side and the right side look the same to me except some subtle differences in the color of shadows…. And another waitlist? Nein Danke.
How can you in good conscience charge a subscription for this? It would be one thing if the app relied on infrastructure, but it's a one and done algorithm that's entirely local on device!
Subscriptions are charged because the developer needs to maintain the upkeep and cost of running the service, but that's not applicable here. It's capable of taking a million photographs fully offline, without any reliance on an online service.
Essentially you're saying your product is so superb that people should pay you indefinitely for the privilege of using it. Even though it costs you nothing to maintain, you somehow need people to perpetually pay you for a local, offline algorithm that you wrote one time. How gross, that's the definition of pointless greed.
Shame on you, honestly. This is exactly what's wrong with modern software nowadays. Just like Adobe and all these other companies, taking advantage of people to milk them dry forever. You'll lease a product, you won't have the right to own anything, and you'll be happy about it.
I'm confused. So this is just producing a single photo that is clearer in a middle rectangle and blurrier outside that rectangle?
Of course the extra detail in the middle is nicer, but won't the edge of the "detail rectangle" be extremely obvious and distracting? Like when you zoom into satellite view on Maps and it's clear that there's a blurry area and a sharp area?
I don't understand why you'd ever want these two things combined into a single image. I can understand why you'd want to snap photos with both lenses simultaneously as save them as separate photos, but I don't understand why you'd combine them.
Very cool stuff, I've long thought there must be some fancy ways to get extra information for better image resolution by taking advantage of the other cameras.
That said, I must also say that most of the side-by-side comparison photos shown here are somewhat impressive, and in some cases, the photosynthesis one just looks worse to my untrained eye.
_______________
I'll choose to take this as evidence of the fact that you're tacking a very difficult problem, and you're also being honest in your comparisons and not cherry-picking too aggressively.
It does not look like the lenses are firing at exact same time. It can be seen more clearly where subjects are moving (woman with her baby in pool for example).
Is that a limitation of operating system? I imagine if it is possible to do that at exact moment, some further processing might be possible
My hesitations on this are it stores data in a proprietary format that I may not be able to export to another common format if the product disappears in a few years.
For something personal like photos and the memories they capture, that's not ideal for me.
While the app is still with us, I'm not sure how much it will cost me to export to my photos to my photo library each month.
I think this is actually how oneplus phones have taken 'night' mode photos for a very long time in their default camera app, it takes multiple exposures and uses the movement sensor in the phone to pick several which were taken with the least hand jitter, and then stacks them into the final result.
This appears to use a naïve masking approach (using ML to find the edges of the mask) which seems to suffer from slight uncorrected differences in lens distortion. There's a fairly obvious feathering around the edges of the detailed area, objects shift quite dramatically, and overall I feel like I'd do a better job of this manually. It is a cool idea, but flawed execution. I would prefer something like using ML to generate a displacement map between the wide angle and the narrow angle parts. That way you can map the higher details down faithfully without having to feather the edges nearly as much or compute any mask. That would also solve parallax which I see also looks unaccounted for.
I'm so confused. Does the dev not know that this is what iPhones do by default?
How is parallax error solved? The two cameras don't share the same optics center.
Nice idea.
On my phone, at least, I had to pintch-to-zoom to 2-3X to see the difference.
And at that size, the “synthesized” version looks much sharper.
You might need more cropping for the mobile version. At 1X I couldn’t tell them apart.
Haven’t checked the desktop version yet
This whole thing feels wrong. OP should be working for a phone manufacturer, not trying to sell this as a subscription. How much is it worth to be able to squeeze a little more visual clarity out of my phone? Maybe $20 maximum? And I feel like I’m on the generous side of the population here. Most people will say $0. But if you’re inside of Apple and you can tell them you can add an average of $1 of perceived value to each of their millions of phones sold each year they’ll be over the moon and you can get a fat salary.
Project Indigo from Adobe does that too https://apps.apple.com/us/app/project-indigo/id6742591546
I want to know what is possible if there were triple the number of cameras on the back of the phone and every single one fired on every photo. If they all focused at different points, the telephotos zoomed at different lengths etc., maybe throw some infrared cameras on there as well, what would be the results when combined?
I believe this is something similar to what Lytro aimed to bring to market.
Lytro, Inc. was an American company founded in 2006 by Ren Ng which developed some of the first commercially available light-field cameras.
This might be a personal preference thing, but in every single example photo on that page, I significantly prefer the colours in the "standard zoom" version of the photo. I wonder if they can figure out how to do this fusion without losing that beautiful contrast the zoom lens brings to the table?
This was actually a purported feature of the Pixel 3 camera. Google said it would be constantly taking photos and the different information captured as your hand moved slightly could be used to create higher information density photos. It didn't work very well at that point but the idea was cool!
very cool app, a few questions: why does the color "temp" of the image change between traditional iphone photography vs your app? it appears to both warm and cool the images. Also why is the contrast decreased, specifically in the library/book store example?
This is interesting. I'm just curious to know how long will it survive until Apple adds something like that to their software. Do you have a fallback plan for this? But I must say that the idea is quite interesting.
Honestly, I assumed the stock phone apps are doing this by default. It would be stupid not to? It's like a low-hanging fruit present ever since phones started having multiple cameras on the back.
(EDIT: though the only real confirmation of this being done that I have is, Samsung's "Camera Assistant" addon has an option to enable "2x zoom" as separate button, and the description there states explicitly it's a synthetic zoom made from combining the main and telephoto camera.)
(EDIT2: come to think of it, for almost any regular mode, if I use my fingers to block two of the three cameras, the image is unchanged, which empirically proves at least Samsung is not doing this - which again makes me wonder, why not? It's an obvious thing to do.)
On that note, I don't understand why there's no widespread, genai-free apps for "super resolution" aka. take a video (or a burst photo shot) of a stationary thing and use it to reconstruct a single photo with much higher detail level.
For videos I can imagine the reason would be that video compression is literally doing the inverse of that (i.e. the information you'd need to reconstruct a higher-resolution image is exactly the one that the compressor throws away for 90% of the frames), so the algorithm and compressor would cancel out. But that doesn't explain why this doesn't seem to exist for burst photos.
I like the idea of this app and the website is well done. Subscription isn't for me, but for folks who take a lot of photos and need this sort of thing, I can see it. Free tier could offer a few more photos per month.
Impressive! And I love the comparison tool on the website.
One more thing should be added as an overlay on each photo are the EXIF data, I want to see that comparison as well.
As an Android guy and not an iPhone, or phone pic guy...
What do the multiple lenses do then, if not this? Like,I presumed that was the point.
Are they LITERALLY just decoration for me to make fun of?
Weird issue: The iPhone dropdown doesn't include the iPhone XR.
It's not like I expect it to work, I'm just curious what the app does.
That demo looks like Minority Report level enhancement. Amazing if it really works, but I'll be damned if I'm paying for a SaaS camera app.
Anyone know if they’re using the native camera resolution for the zoomed part or doing sensor fusion to get better quality than either alone?
I remember that the iPhone 15's official promotion said it would combine 8 or 10 photos into one, right?