▲ 241 points
back
83 comments
A few comments suggest watching the video, but since it's not quite obvious, the video is embedded in this tweet: https://twitter.com/zhuowei/status/1671532250020683787 / https://nitter.net/zhuowei/status/1671532250020683787
Thanks! This made absolutely no sense until the video
I still don't understand it after watching the video. Is the video recorded through a VR headset?
It's a recording of the phone screen running their app. The input of the app is the phone's camera, which they have pointed at their hands holding their Switch. The output of the app is the input plus the game overlaid on top of the Switch.
Oh hey! Nitter is working again! Good to know
What I find very cool is that it's extracting scenes from the running game and then projecting them into the VR space, allowing the scenes to be panned/rotated in a way that isn't possible in the original game.
If you haven't watched the video, check it out; my assumption from just reading explanations here was that it was just projecting a facade of the emulated device and its normal rendering of the game into VR.
The video is wild. Now we need an AI 3D infill to block in all the missing data at the edges of the view.
I actually originally assumed this was entirely done by some AI interpretation of the original 2D projection of the game scene, conjuring the entire new 3D scene. But of course, just extracting the vertex data from the emulator is way better.
I remember all the way back when this round of VR was new I was using a driver called VorpX to reinterpret non vr games into VR.
This Readme doesn't really explain anything. I don't understand what this is meant to do.
It's pretty clear from the title; it's an AR app that projects an emulator over top of your controller.
Sounds pointless if you're using a phone as your window into AR, but if you were to imagine this on a headset like a Quest or Vision Pro, it would basically "turn" any controller into a handheld retro console.
Just watch the linked video on twitter :)
There is a video link, but that link is not very easy to notice. I missed it when I first saw the readme
I think this type of camera control has some big potential in the AR market. It's not fully submersive, but (1) (almost) no nausea because you're peaking into the world (2) screen everywhere (3) can be used in many genres ranging from casual to RPGs and actions (4) feel more natural to existing gamers.
Somehow this made me more convinced with the future of AR, but not VR.
Now that would be a compelling day one Vision Pro app.
It would be, except Apple don't allow emulators inside their walled gardens.
I can load up an emulator on my mac just fine. Whether the Vision Pro allows installs outside of the App Store is still unknown.
It's very unlikely Apple would allow side loading in MacOS if it was released today rather than 22 years ago.
"Walled garden" doesn't describe the Mac platform.
Don't use the marketing phrase 'walled gardens', its pro-corporate marketing jargon.
When has Apple ever referred to their ecosystem as a walled garden? I've only seen it used negatively.
This might work better on the XReal AR glasses that allows for multiple types of input sources.
This seems like an incredibly specific demo. Even they say that it doesn't work with games that aren't mario kart.
Unlike the NES which has only a few ways to layout the display, which the 3D NES emulator relies on to get good results, the DS can do arbitrary 3D stuff. It also has no API layer to reliably extract the camera information to generalize this kind of re-projection.
Neat looking trick but its sort of a dead end. It requires hand written code for any game you would want to do. I'm also curious about the rendering artifacts visible in the demo.
The DS is very much the wrong choice for this. They should try something similar with Dolphin and gamecube games, because the community there already does some stuff with camera manipulation.
There is a fork of Dolphin for VR. I played a few minutes of Zelda with my Oculus on several years ago. Pretty trippy to be transported to a game I enjoyed on a CRT 22 years ago, with a life-scale Link staring me in the face, 3 feet away.
This reminds me that there seem to be few "VR ports" or "cross platform VR releases" of normal games. I guess the gameplay of standard games doesn't necessarily work very well with the inherent limitations of VR. (E.g. nausea from moving around too much)
The NES also had many games with specific chips inside the cartridge, so I guess it is not too unusual for emulators to have game specific code.
There's way fewer NES mappers than there are DS games. It's 30ish vs hundreds. On an NES emulator, coding up just a few mapper chips gets you most of the library and probably 95% of the games worth playing, whereas with this, you need to write a "camera and 3D asset exporter" for each and every title you want to emulate. The work to give this effect to any game is nontrivial. Honestly interacting with the AR library and headset is probably the EASIEST part of this demo.
> It also has no API layer to reliably extract the camera information to generalize this kind of re-projection.
What do you mean?
Why do you have to be so negative about it? The README itself says it's a
"prototype/concept". It's a neat hack, certainly the best thing I've seen on this website today. You make entirely valid technical points, but you don't have to present them as if you were scrutinizing someone's startup pitch. It's not that.
With ARKit you could probably get it to recognise and track the controller using it's object recognition, there's sample code here : https://developer.apple.com/documentation/arkit/arkit_in_ios... Then you wouldn't have to strap the phone to the controller
I'm not so sure about RealityKit, I found it doesn't have the flexibility and finer grained options of ARKit, it's annoying that Apple make 2 frameworks for AR but don't have the same features in both
This is extremely impressive! I was very confused until I saw the video, and then it all made sense.
Excellent work!
It would be cool to see something like this that could handle the 3D slider on the 3DS.
I've been thinking about making a stereogram viewer for my Hololens 1, I think you can make a left and right texture in Unity.
I feel like we are living in the start of a golden age of homemade hacks.
This makes me think, is there a Nintendo 3DS emulator for VR by any chance? Unfortunately non-VR screens which are able to display stereoscopic 3D are getting increasingly rare again.
I was hoping for some polarization and high frame rate trick to allow a passive, reflective screen to show the top screen.
not for DS/AR but for directx VR vorpx did it first.
I mean, that is pretty cool but... It seems kind of obvious that they could have just printed the QR code and taped the paper to the switch instead of using rubber bands to attach a phone to their switch...
having a spare phone is probably more common than a printer these days
Not sure why this is getting downvoted, I thought the same. Having worked on plenty of projects that rely on tracking a QR code, you'll get a much more stable experience scanning it on paper than an LCD screen.
Doesn't the phone control the accelerometer data?
Who needs an accelerometer when you're just interpreting polygon transform/coordinates in a camera view (the camera that recorded the video)
Oh, I thought those came from the switch. Then why not do away with the switch though. Ok, probably just very improvised, quick demo indeed.
SadlyItsBrad demoed something similar
This guy needs to cut his nails tho