[HN Gopher] Enhancing Photorealism Enhancement
___________________________________________________________________
Enhancing Photorealism Enhancement
Author : lnyan
Score : 480 points
Date : 2021-05-12 02:56 UTC (20 hours ago)
(HTM) web link (intel-isl.github.io)
(TXT) w3m dump (intel-isl.github.io)
| nikodunk wrote:
| Amazing how this is all done in real-time, and incredible
| stability of image.
|
| Further, I'd love to also see a tech demo using a Californian
| dataset as the input: To my originally European eyes, it's simply
| making GTA look more like video shot in Germany vs more realistic
| (there's a difference!), though I get that any data set can be
| fed into this :)
| mosselman wrote:
| Is it real-time rather than post processed?
| nikodunk wrote:
| Great point I don't know
| thih9 wrote:
| From the "Enhancing Photorealism Enhancement" video [1]: "It
| is a convolutional network which produces images frame by
| frame and can be run at interactive rates.".
|
| [1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
| dandellion wrote:
| From the paper:
|
| "Inference with our approach in its current unoptimized
| implementation takes half a second on a Geforce RTX 3090
| GPU."
|
| That means 1 or 2 frames per second, so interactive but
| pretty much unplayable, which is to be expected as it's
| just research, but considering that they have DLSS working
| in realtime with tensor cores perhaps something like this
| will also be achievable very soon.
|
| Edit: Out of curiosity I looked up how long DLSS takes to
| process and it's less than 2ms, so you'd probably need to
| speed this up by two orders of magnitude to run it a game.
| L_ wrote:
| There's also a version trained on Mapillary Vistas, which has
| images from all around the world. Results from that are at the
| bottom of the page and at the end of the video (
| https://youtu.be/P1IcaBn3ej0?t=462 )
| janekm wrote:
| To me that version is much more reflective of the actual
| potential of this method than "GTA Dusseldorf dash cam". When
| watching the images produced with that one I get the sense
| that the sense of realism is enhanced by the dashcam-style
| image degradations and could be largely emulated in-engine
| (green LUT, blur + glow filters to emulate lens issues).
| amelius wrote:
| I noticed some strange (unrealistic) effects on the back of the
| red car at the 8:19 mark in this video:
|
| https://youtu.be/P1IcaBn3ej0?t=499
| TGRHavoc wrote:
| You mean the shadow of the tree?
| alkonaut wrote:
| Is there a cert issue? I can't access the page. A cache anywhere,
| or youtube link perhaps?
| himinlomax wrote:
| It seems to be completely removing the atmospheric haze,
| unfortunately.
| dvt wrote:
| At first I was impressed, but then I realize they also work on
| (raw) G-buffers, depth maps, normals, etc. grabbed directly from
| the engine. I mean if we're already doing all of that, why not
| just pass it through 3DSMax and get an _actual_ photorealistic
| rendering? Ray tracing is already starting to be a thing in
| consumer products, so we 're bound to get photorealism at
| interactive framerates in the next decade or so. No neural
| networks needed.
|
| There might also be a misconception that GTAV _couldn 't_ make
| their game more murky/realistic-looking, but more often than not,
| saturation and contrast is _purposefully_ cranked up in games
| (particularly driving ones). I will say that the lush background
| mountains look cool though.
| phromo wrote:
| For an example of photorealistic rendering approach coupled to
| a scenario generator, see
| https://synscapes.on.liu.se/index.html
| virtualritz wrote:
| > [...] Ray tracing is already starting to be a thing in
| consumer products, so we're bound to get photorealism at
| interactive framerates in the next decade or so. No neural
| networks needed.
|
| Define 'photorealism'. You limit yourself to the rendering
| aspect and ignore the content/asset problem and thereby
| economic factors.
|
| This technique is probably one or two orders of magnitude
| cheaper than generating geometry and shaders that have the
| required level of detail in any traditional way.
|
| Either by an artist or programmatic/procedural (Someone has to
| write that code too/set up that node graph in Houdini or the
| like). Yes, you can also just 3D scan stuff (see Quixel, etc.)
| but that has limits too.
|
| More specifically for achieving photorealism by 'traditional
| means': consider scales.
|
| An asset, however produced in finite time, will only hold up to
| some range of scales. The technique in the paper allows to get
| as close or far away as you want and everything will still look
| real. No level of detail work needed, just a few more training
| data.
|
| Traditionally, if you use textures, you will reach the limit of
| the texture resolution. Even if you use procedural techniques -
| not everything is a fractal. A close up of a brick looks very
| different from a wall of bricks etc. Again, who will write that
| code?
|
| My guess is rather that we will see a hybrid of more
| photorealism by traditional means and more 'icing on the cake'
| by methods like this.
| dsign wrote:
| I watched their video, and I tend to agree. With the amount of
| computing they use, ray-tracing is probably going to be just as
| good and probably higher frame-rate. But maybe this technique,
| in due time, will result in an alternative class of rendering
| pipelines with less focus in shading and materials.
| wffurr wrote:
| The Cityscapes dataset is a bit murky looking; try scrolling
| down to the Mapillary Vista dataset instead. Much better
| colors. They should have led with that one instead.
| mrec wrote:
| Not disagreeing, but the Cityscapes samples looked 100%
| photorealistic to me, whereas the Mapillary samples just
| looked different.
|
| Disclaimer: in the UK, so grey and murky == realistic.
| anthk wrote:
| >Disclaimer: in the UK, so grey and murky == realistic.
|
| Ditto in Northern Spain.
|
| The sunny dataset is more akin to Mediterranean landscapes.
| To me the shinny/sunny/reflective stuff it looked unreal.
| ma2rten wrote:
| I think it's worth exploring multiple approaches to a problem.
| Maybe this neural network approach can be improved upon and it
| will also generate photorealistic looking images but have other
| advantages such as less effort on behalf of the game developer
| or less compute.
| calaphos wrote:
| I don't think the goal of this is to make games look better.
|
| Especially the GTAV dataset is often used as a synthetic
| dataset for research. The appeal is that you can extract
| normal/material and other semantic information from the game
| engine for a driving scene.
|
| Basically avoid expensive segmentation labels on real data by
| working the other way around. Things like this synthetic to
| real translation can then be used for other downstream tasks
| (semantic segmentation, distance/normal prediction).
| dirtyid wrote:
| Good ray tracing is expensive and still requires a lot of asset
| preparation. Difference between what RTX can do realtime right
| now and production quality final renders is massive. I think
| there will ultimately be quality wall for realtime tracing that
| target ms/frame versus mins/frame for HQ renders. After which
| hacks like this must be used to further close distance by
| adding layers of additional details that really sell
| photorealism. TBH I doubt RTX could reproduce even this quality
| for many years, if ever.
|
| It's not just about murkiness and lighting quality, this is
| akin to adding the last 10% of polish that usually takes 90% of
| the time. It's also the difference between amateur vs much
| greater proficiency level. Having access to the assets of scene
| in 3D max, someone with a basic training can replicate base GTA
| scene, the enhancements shown here takes years more experience.
| The amount accessibility and saved labour is tremendous. It's
| why even VHS video is still more photo realistic than 4K RTX.
| Real life has a lot of minutiae and interacting details beyond
| resolution and good lighting models.
| dkarras wrote:
| Realistic renders with raytracing assumes accurate models. In
| the scene, the grass can be a rectangular block with a grass
| texture applied to it for instance, NN will transform it into
| an actual grass patch (see example images) similar to how it
| would look like in real life (not constrained by your 3D
| geometry / scene setup). So realistic lighting is just one part
| of the problem space here. A road with an asphalt texture
| applied on it will need a lot of manual tweaks / material
| design to look realistic under raytracing regime, NN can look
| at it and implicitly say "hey, this is a road, and I know what
| a road looks like under this lighting, so I'm going to render
| it like this" and it automatically becomes realistic.
| lloeki wrote:
| Almost sounds like one could grossly placeholder-mark-paint
| surfaces/areas by type and the NN stuff will paint over it
| proper content and consistent details by dreaming them from
| these semantic hints.
| drxzcl wrote:
| You can probably have the game output custom-made data to
| serve as an input for the subsequent NN and skip a whole
| bunch of lighting calculations.
| arketyp wrote:
| Good point. But I wouldn't rule out neural networks. They could
| still be used to derive efficient shader models that can be run
| in real time. https://arxiv.org/abs/1603.06078
| HellDunkel wrote:
| You no longer need 3dsmax for raytracing. But what could be
| really interesting is to train your ml model against path
| traced images and apply the outputs to selective areas of your
| screem where the g buffer provives insufficient information
| density (i. i.e.foliage)
| T-A wrote:
| https://en.wikipedia.org/wiki/Deep_learning_super_sampling
| anticristi wrote:
| I share your disappointment. There was recently a thread on HN
| on how good metadata always beats machine learning.
|
| I wonder how much complexity could be saved by letting the
| application properly annotate objects: "This is a tree located
| at coordinates X,Y,Z in the scene. Please NN take over and
| project a tree appropriate for the light conditions and
| geographical location."
| wefhi wrote:
| "return of GTA IV"
| playpause wrote:
| Uninformed take: this looks to me like an example of 'style
| transfer'. If so, it doesn't seem as compelling as other examples
| I've seen before, eg making a photo look like a Van Gogh using a
| single painting as input, which has been around for a few years.
| Can someone with better knowledge in this area explain if I'm
| missing something?
| L_ wrote:
| Style transfer commonly hallucinates objects or adds artifacts.
| Making something look like a painting is actually much easier
| than making it look realistic. In paintings, artifacts are
| often tolerated as 'artistic'. In a photo, it just pops out.
| Also, style transfer approaches are not temporally stable most
| of the time. Have a look at the comparisons with state-of-the-
| art image-to-image translation (CUT, TSIT) and photo style
| transfer (WCT2). In the case of GTA/Cityscapes shown here, most
| of these methods put trees in the sky and/or flicker. Also the
| use of G-buffers allows a much deeper and more robust
| translation than just using images as style-transfer
| approaches.
| playpause wrote:
| Thank you, this is really interesting.
| [deleted]
| max_ wrote:
| How does this compare to Physical Based Rendering [0]?
|
| [0]: https://en.m.wikipedia.org/wiki/Physically_based_rendering
| 317070 wrote:
| For one, it will be a lot faster to compute. For two, it is a
| lot less laborious work to set up, which does not scale with
| the size of the world like PBR, but only with performance.
|
| So where previously, you needed a lot of designers to make a
| big world look realistic with PBR, now you need to collect a
| big dataset with the style you want to have, and apply it to
| your game. I suspect that will scale differently.
| dahart wrote:
| FWIW, I think it doesn't compare to PBR - these are two
| completely different things, and there's no reason you can't do
| both. Maybe as a simple example of that, it's common today to
| run a NN based denoiser on top of a physically based path
| traced rendering.
|
| This neural network operates on images - input is image and
| output is image. Note that it cannot synthesize an image from
| scratch, it can only take an image and make it look more like
| the lighting in it's training set.
|
| PBR is a technique for taking a 3d scene description and
| synthesizing an image, so for PBR you need models and lighting
| and materials, and even with physically based techniques it
| might still come out not completely realistic looking. There's
| always room to emulate realistic subtleties of texture,
| lighting, camera lenses, video/film response, outdoor colors,
| atmosphere, etc.
| larschdk wrote:
| Impressive, but I think that photorealism is not necessarily what
| you would want to strive for in games, as it is not true realism.
| It emulates the deficiencies of the camera that we perceive it as
| realistic, because we are trained on thousands of photo images
| and video footage. I prefer visionrealism instead of photorealism
| in my games and don't really find that lens effects like flares,
| chromatic aberration, and depth of field enhances the experience
| of realism. It feels like I'm looking through a camera rather
| than being there myself. For me it breaks immersion instead of
| enhancing it.
| captn3m0 wrote:
| This could extend beyond games easily. I can see adoption from
| animation studios for eg.
| dwd wrote:
| I found their approach to be more like the difference between a
| digitally filmed movie (that can look too crisp) and one shot
| on cellulose that has a kind of softness to it.
|
| Where it noticeably fell short was displaying red lights (tail
| lights, traffic lights, etc) that lost their brightness.
| RIMR wrote:
| I don't think you could be any more wrong.
|
| Enhancements like this are going to be the thing that brings VR
| fully into the mainstream. It will be industry-changing.
|
| This tech is in its infancy. I promise you, the final result is
| going to be indistinguishable from human vison.
| larschdk wrote:
| Looking like a photo is quite distinguishable from human
| vision.
| avereveard wrote:
| yeah, games visuals are more about art direction than search of
| realism
|
| this would work great in a flight sim tho, albeit blur and out
| of focus effects are terrible for visual clarity
|
| however, there's one small bit that has not been given enough
| attention imho, which is on the latter set, under "removes
| distant haze": this would be a great technique to complement
| flat LOD reduction of open world games while keeping up the
| crispness and contrast of distant items.
| ginko wrote:
| That's mostly a problem with the input data. A gamestudio of
| the size of rockstar could afford to acquire their own custom
| fit dataset.
|
| Get some good cameras and lenses and capture footage of
| wherever they want to set their next game. Then they can apply
| some motion picture color grading to that and use that as the
| input data for the NN.
| ryankrage77 wrote:
| I think photorealism has the advantage of maintaining a
| cinematic feel. You can do a lot of 'realistic' things withs
| light in a game, that aren't really realistic (you'd never see
| them in real life), but are the type of thing you'd see in a
| photo or movie. This allows for a more tightly controlled
| experience.
|
| Of course, this all depends on what the intention behind the
| game is. If you do want maximum immmersion, visionrealism would
| be the way to go.
| oceanofsolaris wrote:
| I am especially impressed by how much more realistic the lighting
| ends up looking despite the network probably having less
| information to work with than the engine! Though it ends up
| turning the lighting very flat. I guess that is an effect of the
| source images they were using.
|
| As others have said, using this directly in the engine (tuned to
| work with the intended art style etc) could probably produce
| almost miraculous results, if it can be made to work at a
| reasonable frame rate.
|
| That would also allow the developers to use high-quality rendered
| images instead of these green-tinted, low-contrast "automotive
| grade" camera images as source of truth (there are good reasons
| these images look like that ... but they don't look pleasant).
| [deleted]
| lobocinza wrote:
| Photo-realistic burgers
|
| http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food...
|
| They made the game look bad with shitty fps.
|
| If GTA developers wanted they could make the game look photo-
| realistic but that was never the intention.
| anentropic wrote:
| It makes it clear there's way too much contrast in the road
| textures in the actual game.
| PedroBatista wrote:
| Kinda looks like Google StreetView.
|
| Looks great in overcast settings but has some difficulties with
| sunny whether.
|
| Pretty impressive.
| wodenokoto wrote:
| Wow, that is pretty cool.
|
| Maybe GTA 6 or 7 will include no textures and lower poly models,
| and instead spend a large budget driving around LA and just film
| stuff to train neural render.
| xpuente wrote:
| Why is the demo at 720p? Does not look good to do something like
| that.
| rasz wrote:
| There is a long an rich tradition to release your DNN/FX work
| in a post stamp sized presentation.
| dirtyid wrote:
| Impressively more google streetview / Russian dashboard camera
| like. Not a knock, quality improvement noticeable. Base GTA feels
| like intern 3D rendering work, after looks produced by someone
| fairly talented / competent. I can see this saving and incredible
| amount of labour.
| jiofih wrote:
| > GTA feels like intern 3D rendering work
|
| Lol, I guess you're new? Or maybe you don't know the difference
| between 3D rendering software and game engines?
|
| GTA V came out in 2013 and the fact that it is still used as a
| reference for graphics quality tells you everything you need to
| know.
| dirtyid wrote:
| My old dayjob was photorealistic architectural renderings.
|
| The point is GTAV, which was/is exemplar for game engine
| graphics, and even current gen RTX game engine rendering all
| look "bad" due to constraints and budgets of real time
| rendering. Bad in the sense that they're necessarily
| rudimentary - actual photo realism requires many layers of
| expensive details. This technique fills in many of those kind
| of details. The improvement in photo realism demonstrated
| here is significant and a huge leap in terms of image
| quality.
| aspaceman wrote:
| Oh this is really impressive.
|
| I've always wondered how games were going to solve the last mile
| of photorealism, and an additional color-grading filter seems
| like the perfect step. Really impressive work.
|
| I think titling it "photorealistic enhancement" is a bit of a
| whiff, but "photorealistic real-time color grading / filtering"
| is bad-ass.
| orbital-decay wrote:
| They mention temporal stability across the footage, but what
| about the entire game? It would be unfortunate to have the same
| car to turn from Audi to BMW under entirely different lighting
| and angles, or a palm tree into a pine. (the difference might be
| much more subtle but annoying nonetheless)
| dahart wrote:
| Part of the contribution of this paper is about preventing that
| specific problem.
|
| "We also seek to eliminate artifacts that can be seen in the
| results of prior deep-learning approaches, which often
| hallucinate objects. To this end, we analyze the datasets that
| are commonly used for photorealism enhancement. Our analysis
| reveals that their scene layouts differ in ways that can
| explain artifacts commonly seen in prior work. To better align
| the datasets and alleviate the artifacts, we propose a new
| strategy for sampling image patches during training. We further
| design a new adversarial training objective that facilitates
| enhancements that are geometrically and semantically consistent
| with the content of the input image."
| bshanks wrote:
| Many of the GTA V pictures look more realistic to me. I think
| some of this is due to location; in the GTA V pics, the brownish
| hue, the haze and washed-out colors, the patchy grass, and the
| grainy roads seem to me to really capture the actual look of
| typical southern California urban sprawl. The enhancement makes
| things greener, less hazy, more grassy, and smooths roads, which
| is more typical for many areas of the United States, but not for
| southern California.
|
| In addition, the enhancement blurs a bit, which imo is not good;
| the blur adds to photorealism by hiding mistakes, but on the
| other hand it makes it more obvious that you are looking at a
| picture because you can't see details.
| yummypaint wrote:
| For me the most unrealistic part of the GTA renders was the
| trees, and the post processing significantly improved that.
| Blur is also an important part of real photos, it's unusual for
| everything to be in perfect focus.
|
| I would argue that these kinds of renders are getting good
| enough that we should be more precise about what is meant by
| photorealism. Most pictures i take with my phone have alot of
| blur, don't capture colors well, and are frankly quite crappy.
| I feel like people expect photorealism to be equivalent to a
| really well-planned photo by a professional.
| anigbrowl wrote:
| I mean it just looks like they're simulating the windscreen
| glass. I guess it's more realistic but I somewhat prefer the
| original.
| StavrosK wrote:
| Is the windshield glass really the only change you see, when
| grass becomes green, reflections are added, the entire photo
| has a higher color temperature, etc?
| caslon wrote:
| Most of it is making light more intense, and most of
| everything else could be emulated with a blur filter and a
| green tint.
|
| It's impressive for what it represents technically, but it's
| visually subtle.
| derefr wrote:
| Looking at the last comparison on the page, the grass isn't
| just becoming green, it's being "puffed" out from its
| original geometry, as if the grass geometry is being vertex-
| shaded with small spiky-ball-like objects. (A bit like the
| look of "static grass" in model making.)
| rzzzt wrote:
| I think the Cityscapes dataset pictures were taken from the
| car's hood; the Mercedes emblem is up close. The camera
| position in game also seems to match this.
| Balgair wrote:
| That's really cool!
|
| But why does every car look like it's been freshly waxed? The
| newspaper boxes have a bit of rust and grime on them. Those look
| really real.
| serf wrote:
| is the road-smoothing an artifact from the fact that the camera
| input sources lack the ability to resolve a detailed picture of
| the road at-speed?
| 317070 wrote:
| Probably yes. Alternatively, it shows that the road textures
| used in GTA V were not very apt for their job. I do think the
| smoother ones look considerably more realistic.
| stanfordkid wrote:
| I would love to see this on NBA2K21 ... seems like you have an
| ideal training set with recorded basketball games.
| cousin_it wrote:
| Nice! So a next gen game could be made with a simple renderer,
| painting areas of the screen with a tag saying "grass" or
| "window", and a neural network trained on high quality renders in
| the desired style (not necessarily photorealistic). The same
| technique would probably work for character animation as well.
| neonological wrote:
| One day there's going to be a gan where the input is a random
| number and the output is the source code of an entire triple A
| game.
| BiteCode_dev wrote:
| Very impressive. It will hard to balance realism and readability
| though.
|
| GTA uses some surrealist contrasts because it makes the games
| easier to play, especially for long hours, or when you go fast.
|
| But the tech is so new I'm not going to complain, it's already
| very good, especially given what it eats in resources.
| ohlookabird wrote:
| Looks really nice. It is however funny to see what it does to
| road signs (black arrows suddenly turn blue or the orange "don't
| walk" hand of pedestrian traffic lights seems to disappear). I
| wonder if this has to do that they use image data from Germany
| (having slightly different signs than the US).
| zuzun wrote:
| It's also giving the license plates the blue stripe.
| bawolff wrote:
| The results dont look that great to me. The original looks better
| for a game. The "enhanced" version is so unsaturated it could be
| bordering on black & white.
| ObserverNeutral wrote:
| germany is unsaturated
| eutectic wrote:
| The model trained on TSIT gives nicer results.
| svantana wrote:
| That's because of the dataset. Later in the video they show
| results using another dataset with more vibrant colour, that
| looks a lot better IMO.
| bawolff wrote:
| I just watched the video to the end - definitely an
| improvement, but it still looks "faker" than the original to
| me. Maybe some uncanny valley effect is going on with me.
| jiofih wrote:
| You're just used to the style. Putting a photo of a road
| next to it will show you how far from reality GTA rendering
| is.
| 317070 wrote:
| Out of curiosity, did you play the game? To me, it is hard
| to imagine someone would find the game to look less fake.
| bawolff wrote:
| I did not. I'm not saying the game looks real, just that
| it looks less "wrong" somehow, to me.
| gonzo41 wrote:
| This is pretty impressive, I wonder if you could build really
| cheaply textured maps and then upgrade them with this method as a
| post production touch up. Essentially removing most of the human
| work from texture libraries.
| [deleted]
| micheljansen wrote:
| Really impressive and striking to see the stylistic effects of
| different training sets (Cityscapes vs Mapillary Vistas).
|
| Obviously the GTA V designers applied a particular style and
| colour scheme which gets lost, but I'm sure something like
| StyleGAN could address that.
| Gauge_Irrahphe wrote:
| Why not let it output the gbuffer images instead, or perhaps even
| let it operate on the artwork instead? That way you could bake
| the changes into the game without having to run the NN in real
| time.
| dahart wrote:
| This is a good question - it certainly is possible to run a NN
| on the artwork or pre-g-buffer, and you could imagine having
| something bake the changes.
|
| To do that, you'd need access to the game's source code and
| assets, and you would need to rewrite portions of the rendering
| engine. Training might be incredibly slow if it needs to
| iterate on builds of the game, or that problem could be worked
| around but might need considerable engineering to support the
| training iterations (e.g. data representation and a pipeline
| with the ability to render changes to the artwork and renderer
| on the fly without needing a game rebuild).
|
| So I'd speculate the answer to why not is that it's just not
| what the authors set out to do, that they don't have access to
| the game source nor the time or resources needed to refactor
| the renderer.
|
| Part of the magic of neural networks is their black box
| properties - they can do what they do without needing to
| understand 3d geometry or integrate with a render engine. Throw
| an image in, and a new image pops out, without the neural
| network needing to understand what it's doing or why.
|
| That said, I would guess that coming down the road is examples
| of exactly what you're suggesting - games that will use neural
| networks to drive realism in the assets and shaders and
| renderer. To some degree there are already tools starting to do
| this.
| derefr wrote:
| From what I can tell, the whole reason the road "looks like a
| video game" is that it's using a source texture and then
| stretching it along the road's geometry, "diluting" the level
| of detail as seen from the camera viewpoint.
|
| The image enhancement here is happening "in camera", like a
| fragment shader; but it's using a lot more context than the
| sort of local/cellular kernels that fragment shaders usually
| are.
|
| Look at e.g. the gutters on the roads, where there are
| leaves/dirt -- they get filled in with tons of extra
| _perceptual_ texture (texture that exists in the projection
| plane, rather than texture that exists "on" the road),
| bringing them up to the same LoD as the rest of the road.
|
| If that texture was "pushed back" to the origin texture, said
| texture would have to exist at some sort of ridiculous
| resolution that modern graphics cards wouldn't have a hope of
| rendering.
| gmueckl wrote:
| The part about ridiculous resolutions for texture detail like
| leaves isn't necessarily true. There are very old techniques
| for scattering small decals densely across surfaces. The
| basic trick is to use coarse textures that encode the
| positioning of decals within each texel footprint.
| derefr wrote:
| Sure, but no matter what the geometry/bump maps/etc. are on
| the road, they're not going to render out as anything but
| an aliased, badly-lit mess unless you're heavily
| supersampling them (and then doing ray-tracing to the
| supersampled image.) We're talking about "objects" where
| the crucial detail-features of their meshes for getting
| correct color math out (e.g. the normal+reflectance maps)
| would be, on average, scaled to be smaller than a pixel
| from the viewing distance+angle, with potentially many
| objects contributing to the final color of each pixel.
|
| So, rather than rendering the _road_ at 8x-16x and then
| downsampling it, in order to get a pile of leaves to appear
| to have ~1.5x LoD at the projection plane, you can just
| achieve that 1.5x LoD pixel by pixel _at_ the projection
| plane.
|
| Or, to put that another way: when a digital artist is
| painting a landscape, they don't have to zoom in by 8x,
| paint every individual blade of grass, and then zoom out.
| They can just use an artistic technique at 1x zoom that
| approximates the texture that would have been created by
| downsampling "photographic" detail painted in at 8x to your
| screen resolution.
|
| Here, the ML model is playing the role of the digital
| artist, "painting" on the projection plane.
| enriquto wrote:
| The problem is not texture antialiasing (which is
| essentially solved). The problem is _shape_ antialiasing,
| whether many distant small shapes are so close together
| that become a sort of translucent buzz, which is very
| computationally heavy to compute.
| gmueckl wrote:
| REYES renderers had a pretty nifty solution for shape
| antialiasing with their slicing and dicing of geometry
| into sub-pixel sized primitives. That was in the 80s,
| just for reference. Unfortunately, this method has a
| pretty complex implementation and just doesn't play all
| that well with hardware rasterization or path tracing and
| has since fallen out of favor.
| Gauge_Irrahphe wrote:
| I don't think so. I think it's because the texture is already
| photo-like, and then there is a normal map applied on top of
| that. It basically creates two layers of shading that don't
| fit together. Few games manage to avoid that problem.
| derefr wrote:
| Interesting hypothesis. I think this would predict that
| you'd get more realistic results if you were using "bare,
| unlit" textures (e.g. texture images + normal maps created
| by using 3D scanning or even computational tomography, as
| you would if you wanted to 3D-print a copy of the object)
| and then applying regular normal maps to them. Anyone have
| screenshots of 3D game objects that were explicitly
| mentioned as being authored this way?
| jeisc wrote:
| Does any one here know about some existing self-driving(self-
| crashing) technology that uses two cameras in unison on the sides
| of the car like the eyes of a equide providing a nearly 360 angle
| image only readable by the computer due to 2D perspective
| constraints.
| Toutouxc wrote:
| So, can we gradually get to a point where the engine itself only
| renders simplified abstractions (i.e. not the current state-of-
| the-art graphics) and feeds them to a neural network that
| produces the output?
| [deleted]
| arketyp wrote:
| This is what has been called Deep Shading:
| https://arxiv.org/abs/1603.06078
| plutonorm wrote:
| The number of comments here that are negative is just staggering
| to me.
|
| These images are within a hair's breadth of being
| indistinguishable from reality. It's an enormous leap. An
| enormous leap from the previous techniques and an enormous leap
| towards photorealistic realtime graphics.
|
| You don't want your games to look realistic? Go play pacman or
| outrun.
|
| You want to use path tracing instead? You have no idea how much
| cheaper this technique is. And on top of that the neural network
| is also fixing up the unrealistic texture work on all kinds of
| things in the scene.
|
| The first games to use this are going to be from small game
| studios with less investment in the status quo and they are going
| to blow the competition out of the water.
| andrew-dc wrote:
| I think photo-realism in games is not merely a black and white,
| positive or negative. There are trade-offs, and I think it has
| to do with perception around aesthetics.
|
| (also, this is simply one guy's opinion, but I am curious if
| anyone else feels the same)
|
| For instance, in my view, I tend to prefer a stylized less
| hyper-realism in games. I think this is because the enhanced
| color, light, and brilliance is more aesthetically pleasant to
| look at, even though it is certainly more uncommon in the real
| world.
|
| I don't believe this is because saturated color, and brilliant
| light doesn't exist in reality, but these occurrences happen
| more rarely and in specific, often planned, situations and
| settings. And some people (trained or gifted photographers, for
| instance), have learned how to capture this look, or to take
| something rather ordinary, and capture it to appear
| extraordinary, much like the difference between your average
| person snapping a photo with a disposable camera, and Ansel
| Adams. The former will produce a photo that is far more common
| in appearance, and thus...boring, or uninteresting. So when it
| comes to games, if you simulate the "average look of reality"
| the game certainly appears more photo-realistic, but may also
| be less aesthetically interesting (to me, at least).
|
| This says absolutely nothing about the technology developed
| here, which is incredibly impressive. Nor do I think there is a
| right or wrong way create game visuals. I think that misses the
| point.
|
| However, I feel I'm not the only person who can be blown away
| with how realistic something can be made to look, yet ...also
| find it more aesthetically drab and boring to look at. And in
| the case of games, as I mentioned above, I tend to prefer to
| see things that are rare, interesting, and extra-ordinary, and
| thus less "realistic".
|
| Anyone else feel the same?
| amelius wrote:
| Photorealism shouldn't be the end goal. Watch a B/W television
| for a week and you won't be missing the colors. Photorealism is
| much like sugar in that it is addictive, but you won't miss the
| sugar if you didn't eat it for a while.
| Cthulhu_ wrote:
| It's not the end goal, but it's definitely an aesthetic
| direction that a lot of games want to go towards. And not
| just video games, but films too. Most films already have
| photorealistic CGI, sometimes applying it without you even
| noticing it (e.g. backgrounds).
| anthk wrote:
| Better, play IF games/roguelikes such as Slashem/DCSS,
| Gearhead, Anchorhead, Vicyous Cycles, Slouch over Bedlam and
| All Thing Devours.
|
| Then, every AAA game looks dumb, non-interactive and limited
| even with HQ graphics.
| klmadfejno wrote:
| And yet, the vast majority of DCSS players choose tiles
| over ASCII. Liking games without nice visuals doesn't mean
| people don't like nice visuals.
| armageddon wrote:
| > You don't want your games to look realistic? Go play pacman
| or outrun.
|
| I can't just play GTA 5 as is?
| kfarr wrote:
| Yeah the foliage is remarkably life-like and that is so hard
| and labor (art) intensive to do well. This is an amazing tool
| for indies without tons of resources.
| HellDunkel wrote:
| Maching learning is often oversold as a shortcut solution for
| some very hard problems and this is a perfect example.
|
| What you predict about small studios seems unlikely as ML is a
| beast to tame, needs highly specialized engineers and huge high
| qualty datasets for training.
| plutonorm wrote:
| So were ajax crud web apps, 25 years ago.
| zokier wrote:
| And they still suck, 25 years later
| Cthulhu_ wrote:
| Could they do this process at video game speeds though? (25-60
| times a second)
|
| I think this technique has a lot of potential, but will
| probably need a lot of work before it can be applied to video
| games. It might first show up as a filter for 'photo mode' in
| some games though, that would be interesting.
| cecilpl2 wrote:
| The first video on that page says they can run it at
| interactive framerates.
| cycomanic wrote:
| I completely agree with you. Also those people who complain
| about the "dull european" look, did you scroll down and look at
| the images altered using the Mapillary Vistas set?
|
| The cool thing is that this gives game-designers an extremely
| powerful additional tool. You could for example easily go for a
| more retro-look by training with kodakcolor images ...
|
| And I don't buy the argument that GTA doesn't look more
| realistic because that's what the designers were going for. If
| this was a game with a clear "comic appearance" that might be
| true, however I would argue it looks like it does, because the
| designers did the maximum that they could achieve with their
| tools and within a certain budget.
| tallanvor wrote:
| I almost clicked away from the the article because those
| Cityscapes transformations looked worse in my opinion - just
| washed out and dull, so I can understand why people didn't
| like the look. The Mapillary Vistas processing definitely was
| nicer, although it looked uneven to me - the road still
| looked washed out, which created an almost disturbing level
| of contrast with the greenery.
|
| What they've done is interesting, and I think with further
| refinement it is certainly a technique to consider.
| Cthulhu_ wrote:
| Definitely; I mean if they had a dataset of the cities and
| locations GTAV was based on I think it might look uncanny.
| marcodiego wrote:
| I think expectations influenced by culture play a role
| here. Since I live in a Brazilian metropolis (4mi
| population), the Cityscapes transformations looked more
| real. That is probably because they look more like what I'm
| used to see when I walk the streets. People from rich
| neighborhoods in the US will likely find the Mapillary
| Vistas more realistic since that is what they used to see.
|
| Impressive result nevertheless. Improving this with
| adversarial networks will probably generate images which
| are indistinguishable from reality in just a few years.
| formerly_proven wrote:
| The color grade in GTA is of course an artistic choice
| and rendering the entire game with foggy overcast light
| and a strong green tinge (which btw. is not how German
| cities look, we aren't living in a bad color transfer of
| The Matrix) makes it look bad at the macro level. The
| details might be enhanced by this method, but honestly
| it's kind of hard to tell from a heavily compressed 720p
| Youtube video. What looks okay or even better in a video
| doesn't have to look better interactively, uncompressed,
| in native resolution.
|
| Hot take: The overall tone of the style produced using
| this dataset comes from capture artifacts.
| 411111111111111 wrote:
| the only thing i could get out of the demo videos is that
| their fps is shit.
|
| fps > beautiful images until you get at least 60... which
| this wasn't.
|
| im pretty bearish on this kind of innovation in games
| tbh, its never going to be low latency enough to be
| viable, even if they'd improve their framerate - i dont
| see it going beyond academic interest because of that
|
| definitely interesting to read about though, even if i
| doubt its viability in gaming
| nearbuy wrote:
| The Cityscapes transformations have a photorealism
| advantage because they're blurrier and more washed out.
| It's easier to match less detailed photos.
|
| The blurriness, green tinge, and washed out colours are
| qualities photos sometimes have, but it's not what we see
| with our eyes. But they do help hide details that would
| give the image away as a computer rendering.
| bayindirh wrote:
| Actually, I'm pretty amazed by the results. This is indeed a
| giant leap. Also it's coming from Intel instead of AMD or
| Nvidia, which makes me hopeful about its wide availability.
|
| The only thing I didn't agree is that Rockstar is _incapable of
| improving their engine further_. Since GTA-III, this is
| somewhat a deliberate stylistic choice up to a certain point,
| like how Source engine is also opinionated about how its games
| look.
|
| Also, we need to keep in mind that this method is way more
| surgical than a simple post-processor. They've probably grafted
| a lot of tapping points to video driver to be able to implement
| this.
| garaetjjte wrote:
| >They've probably grafted a lot of tapping points to video
| driver to be able to implement this.
|
| They didn't mention using anything else than output frames
| and G-buffers.
| bayindirh wrote:
| I'm not an expert in graphics programming but I don't think
| that accessing G-Buffers of an application and grafting
| your ML based post processor into the pipeline is exactly
| trivial. From my point of view it's only possible by either
| engine provided tapping points, or you tap the processing
| pipeline at the driver level directly.
|
| I'd love to be corrected if I'm wrong, BTW.
| garaetjjte wrote:
| You need to hook into game process, but it's nothing too
| hard. Either patching game binary in right place to
| funnel pointer to needed buffers to your DLL, or hooking
| something like D3D11 OMSetRenderTargets and doing some
| clever guessing which one of buffers this is. Doing this
| at driver side wouldn't make any difference, as driver
| doesn't have any extra semantic info what the buffer is
| used for anyway.
| formerly_proven wrote:
| > The only thing I didn't agree is that Rockstar is incapable
| of improving their engine further. Since GTA-III, this is
| somewhat a deliberate stylistic choice up to a certain point,
| like how Source engine is also opinionated about how its
| games look.
|
| RDR2 is based on further development of the GTA5 engine and
| is arguably the best looking open world game at this time.
| draw_down wrote:
| Whenever there is a sanctimonious comment like this at the top
| complaining about all the negative posts, I usually agree with
| the other posts. As in this case.
|
| I actually don't think this looks any less "like a video game",
| but its color is more washed-out, and darker. It seems like a
| net loss to me, sorry. Not for fancy theoretical reasons, I
| just think the GTA looks better in all the examples.
| codeflo wrote:
| > You don't want your games to look realistic? Go play pacman
| or outrun.
|
| That's a silly dichotomy. We don't even want _films_ to look
| completely realistic, that 's why CGI exists in the first
| place. Imagine that you ran this over something like an
| Avengers movie, or even better, Blade Runner 2049. The AI would
| remove all the stylized/unrealistic details, and someone here
| would claim that this is clearly an improvement, and any
| criticism must come from luddites!
| saiya-jin wrote:
| The primary reason why CGI is used is to save money. The
| secondary is to ease production of movies, which at the end
| saves money and time (which is money).
|
| Styling as done in some of them is an optional bonus if
| creators strive for some additional artistic message, but
| otherwise its all about money, money, money.
|
| You even contradict yourself - they tried to look Avengers as
| realistically as possible within the realm of comic book
| fairy tale, that's why they changed character of Thanos
| significantly well after introducing him (arguably a good
| choice). From cartoonish styling to realism. Nobody would
| take the epic battles so seriously if they all looked like
| Ready Player One.
| klmadfejno wrote:
| If I want to film a dragon, I'm probably going to choose
| CGI because it gets me the best dragon. It is
| coincidentally cheaper than building a puppet, or
| genetically engineering a lizard to look like a dragon, but
| cost isn't the motivating factor here.
| Hard_Space wrote:
| The only director who laboriously follows through on 'I want
| it to not look like CGI' is Christopher Nolan. Others seem to
| favor stylized beauty over absolute CGI realism.
| thenoxx wrote:
| Uh, no, you just don't notice the absolute gobs of CGI used
| in tons of films because it is meant to look
| hyperrealistic.
|
| There are several, several youtube videos showcasing the
| incredible work of studios like Industrial Light and Magic
| creating most of the backdrops you thought were shot on
| location.
| solotronics wrote:
| I thought the video was from real life at first until I
| listened to the explanation. It's amazing that this is possible
| in real time.
| alkonaut wrote:
| > You don't want your games to look realistic?
|
| The creators aren't necessarily aiming for photorealism. They
| want their games to look good enough and be entertaining.
| Adding more realism could make it look worse. Obviously if AI
| could solve the _hard_ problems like global illumination that's
| great. And I'm sure it'll be possible to e.g easily add more
| variation to textures by using larger datasets. But changing
| the tonemapping to a colder palette, or just replacing the
| asphalt textures with smoother ones?
|
| The creators of GTA didn't use the sunset tones in the game by
| some creative mistake or lack of computing power. They
| obviously went for non-photorealism deliberately! It's a
| stylized view of California. It's like the burger in the
| commercial doesn't look like the real one - it looks a bit
| unrealistically perfect because that's actually more attractive
| than the real thing. Realistic and photorealistic are
| tangentially related.
|
| The comparison here should perhaps be between frames of the
| game that already attempted to look like the cityscapes
| dataset.
|
| GTA obviously doesn't have a ground texture that isn't smooth
| because it wouldn't be possible, or "too warm" palette because
| it's too computationally expensive to have the bleak cityscapes
| tone mapping.
|
| A GTA frame rendered at its current frame cost, with textures
| the same size (but different) and a customized tone mapping,
| would look more photorealistic and make the difference to the
| cityscapes dataset much much smaller.
|
| The authors could likely have done this comparison since they
| probably have the ability to modify textures and shaders to set
| tonemapping. That would have separated out the differences that
| are only _artistic_ from those that are lacking photorealism
| because of computational expense.
| kaibee wrote:
| Honestly my main take here is just how much better the
| foliage can look. Grass/trees in real time graphics just
| aren't even close.
| [deleted]
| dplavery92 wrote:
| I am not very interested in whether or not the dataset that
| these authors chose is a pleasing dataset to use as a basis
| for a video game aesthetic. I _am_ interested in in the
| technical accomplishments that this model was able to achieve
| in regards to preserving image features of semantic
| importance (e.g. writing, symbols, logos), not
| "hallucinating" new image features, and maintaining frame-to-
| frame stability in the output video product. These are very
| notable accomplishments for an adversarial model in this
| space, and the techniques they used to accomplish them are
| interesting. That Rockstar could have made their game more
| Cityscapes-like without this tech is missing the forest for
| the trees.
| ksec wrote:
| >These images are within a hair's breadth of being
| indistinguishable from reality.
|
| This. I thought those were photos they took somewhere that
| looks exactly the same as GTA V, but then when I watch the
| video explaining it I was like holy crap. No Ray Tracing or
| whatever tech demo was ever this close to photo realistic.
|
| >You have no idea how much cheaper this technique is.
|
| Could someone explain this in greater details? I suppose you
| will still need to create some decent base image or graphics
| for the enhancement to work to its maximum?
| 0-_-0 wrote:
| I think the source game could be lower polygon count and
| texture resolution, but the neural network might be quite
| heavy...
| ryankrage77 wrote:
| > These images are within a hair's breadth of being
| indistinguishable from reality
|
| Without a comparison to the original pictures, they do look
| very good, the roads in particular look more realistic, as
| though taken from a dashcam or medium quality phone camera.
| However the foliage just looks blurred.
|
| A lot of (interactive) CGI already looks more realistic than
| this. Take for example, this Australian scene made in Unreal
| Engine 4 - https://www.youtube.com/watch?v=Pg75bfkegtU
|
| And that's not just pre-rendered, here's someone exploring the
| same scene interactively -
| https://www.youtube.com/watch?v=y87QL7IPE34
| choeger wrote:
| While I agree that the quality of this method is impressive,
| there are two caveats:
|
| 1. We do not know the set of pictures that the authors did not
| show us. So maybe this technique works really well only for a
| small fraction of cases.
|
| 2. Games (especially games that involve moving cars) tend to
| simplify mechanics so much that the _moving_ end result will
| never look realistic, regardless how perfect the still picture
| is.
|
| So yes, I expect great outcomes from such a method, but I don't
| think its going to be a revolutionary silver bullet.
| malka wrote:
| > You don't want your games to look realistic? Go play pacman
| or outrun.
|
| I want my games to look cool. I dont wanna them to look like
| streetview or geoguessr.
|
| The result is impressive, but I'm not sure it would be good for
| video games. Maybe with some added filters it would be a lot
| better.
| snorkel wrote:
| This is an enormous leap. Texturing and lighting are two very
| hard problems in CGI. Realistic lighting using raytracing
| requires a lot of radiosity passes, which means adjusting
| lighting of each surface to account for each ray of light
| bouncing from surface to surface to surface to surface like a
| 3D billiard ball, more bounces computed yields better results
| but there's computational limits to the number of reflections
| per ray, and many parameters and formulas to choose from each
| with its own trade offs. Texturing and shader pipelines is
| another matrix of trade offs to get a good-enough result.
| Instead of all of that complexity, applying ML this way
| cleverly hops over the most difficult parts to a really
| impressive result.
|
| Some don't want games to look this way. Nonetheless this
| technique can be applied to give games and movies a different
| look that would very hard to acheive by standard rendering
| pipelines, and it doesn't have to look washed out like this
| example, that's all based on the training data set and how much
| this ML lighting and texturing is blended into the scene, and
| that's another creative choice. I can imagine filmmakers
| wanting to adjust CG lighting and texturing of a scene using a
| training data set from old TV shows to make a convincing reboot
| of a classic TV series, or new series done in that style.
___________________________________________________________________
(page generated 2021-05-12 23:03 UTC)