Implementing hybrid ray tracing in a rasterized game engine

Share on linkedin
Share on twitter
Share on facebook
Share on reddit
Share on digg
Share on email

For the past month, many of you have been asking on social media or in the comments section of our blog for a video recording of our hybrid rendering demo. Since some of our readers have missed our GDC 2014 sessions or didn’t get a chance to drop by and see our PowerVR Ray Tracing demos running at our booth, I have asked my colleagues in the PowerVR Ray Tracing group to put together short clips that present the work they’ve done to highlight some use cases for ray tracing.

The first example we’ll be analysing today is the hybrid rendering demo. Before we begin, there is a caveat: please keep mind that this demo is running on a previous generation of the PowerVR Ray Tracing technology. The PowerVR GR6500 ray tracing GPU we’ve announced at GDC 2014 is designed to be integrated in a System-on-Chip and therefore can provide much better raw performance (a higher ray budget) and tighter/faster memory coupling while consuming lower power. Therefore the performance, framerate and resolution in the video isn’t representative of the PowerVR GR6500 or any other Wizard GPU.

The hybrid rendering demo running on our PowerVR Ray Tracing IP, as shown at GDC 2014

For this ray tracing demo, we’ve started with an experimental version of the Unity game engine and implemented OpenRL support which enables us to use the G-Buffer from the rasterizer to initialize rays (see my recent Gamasutra article for more details). Then, the results of the ray tracing are used by Unity’s final lighting pass and fed right back into the game engine.

Ray tracing alongside rasterization opens up many possibilities and enables effects that are either difficult or impossible to do without artefacts using raster techniques alone.

Hard and soft shadows

The first scene presents hard shadows. We are using the properties of the G-Buffer (world-space positions, normal and material IDs) to cast shadow rays. Zooming in on the tree in this scene reveals how every leaf is casting and receiving a very accurate shadow.

Hybrid ray tracing in a rasterized game engine - hard shadows (2)Ray tracing can be used to generate very accurate hard shadows

The second example in this scene introduces a similar situation where the complex geometry of the bridge will cause resolution issues when casting shadows in a traditional game engine. However, ray tracing doesn’t have any resolution problems because the rays are cast for each screen pixel.  Developers do not have to worry about selecting the appropriate shadow map resolutions for various positions on the game level, and rigorously testing the game looking for artefacts. Ray traced shadows look realistic from afar (check out the beginning of the scene) and maintain fidelity all the way to the end. We can go as close to the wall as we want and we will never run out of shadow map resolution.

Hybrid ray tracing in a rasterized game engine - hard shadows (3)Ray traced shadows will not have resolution issues

The next scene focuses on soft shadows. While hard shadows simulate sunlight on a clear day at noon, soft shadows appear when the light source has area (e.g. a lamp with a shade) or there is a medium to scatter the light between the light source and the scene (e.g. clouds). In our hybrid rendering example, by simply adding some random noise to the shadow ray directions, we can obtain the appearance of a cloudier day. This technique simulates the effect where the penumbra size is dependent on the distance between the occlude and the object receiving the shadow. In the video example, the head of the lamp has a very soft shadow while the base retains its hard edges, just like in real life. This would be very difficult to achieve using conventional techniques.

Additionally, implementing ray traced shadows in game engines enables you to have many shadows casting lights because sampling decisions are made on a pixel-by-pixel basis. In the scene above, half of the pixels don’t need to sample the headlights of the car because those lights can be rejected by cheap cone-test. There is no need for a separate pass to handle each dynamic shadow casting light.

Reflections, refractions and transparency

Another use case for ray tracing is reflections, which can be handled in a similar fashion to shadows. Instead of directing the ray towards a known light source, we send the ray in the direction calculated by performing the mathematical reflect function between the vector from the viewpoint and the normal vector at the surface. Ray traced reflections have several advantages over raster techniques like reflection maps.  Not only do they reflect dynamic objects without needing separate rendering passes, nearby objects and self-reflections are handled perfectly.

Hybrid ray tracing in a rasterized game engine - reflectionWhen using ray tracing in your game engine, reflections look crisp clear

Additionally, transparency is also handled by casting rays. In our example, the windshield is tinting the rays and continuing them on to provide order-independent transparency. If you look closely at the car, you can notice we have implemented both transparency and reflections techniques for the windshield glass material; this is easy to do and produces a realistic effect where the trees are visible in the glass. The proportion of the blending between the reflection and the transparency takes into account the Fresnel term calculated from the viewing angle, and produces a convincing model of real life.

Another example of ray traced transparency is the fence. It combines transparent objects with shadows to achieve an interesting visual effect. The fence is made of quads with an alpha texture applied to create the wire mesh. Because the rays respect the texture’s mip level, the shadow just looks like a faint tint from a reasonable distance. Once we get closer, the texture is starting to come out and the shadows gradually become more detailed, right up to the resolution limit of the wire mesh texture itself.

Hybrid ray tracing in a rasterized game engine - hard shadowsIn ray tracing, you can combine transparent objects with shadows

A texture can also be used to tint a ray traced shadow for a quick and easy way to get perspective textures. In our example, this effect is noticeable on the pavement as the light cast from inside the diner is combined with the window decal to produce a coloured shadow.

Finally, this wouldn’t be a ray tracing demo if we didn’t show some refraction. The open sign visible through the neck of the bottle or the bottle’s label is visible through the mug. We apply the refraction equation to bend the ray based on the difference of the index of refraction between glass and air, and then apply it again in reverse when the ray exits the material.

Hybrid ray tracing in a rasterized game engine - refractionRefraction and glossy reflections

Glossy reflections also work with this hybrid method. The normal map on this table means that the reflection rays are scattered a little bit. We can faintly see the reflection of the open sign and the beer label in the table, although it is too fuzzy to make it out clearly.

Final words

In conclusion, we’ve seen how adding just a few rays-per-pixel to a rasterized game engine can add a lot of effects that would be difficult or impossible to do with rasterization alone. Ray tracing can do a better job approximating the physical light transport phenomena, and using ray traced effects can avoid the artefacts of complicated raster techniques and often work out to be even be more efficient.

Thank you for watching! I hope this article has come a little way towards quenching your thirst for ray tracing demos. If you have any suggestions or questions, please don’t hesitate to leave a comment in the box below.

Come back to our blog for a second article where we focus on another example where PowerVR Ray Tracing is used inside the Unity 5 lightmap editor and make sure you follow us on Twitter (@ImaginationTech, @PowerVRInsider) for the latest news and announcements on PowerVR and ray tracing.

Alex Voica

Alex Voica

Before deciding to pursue his dream of working in technology marketing, Alexandru held various engineering roles at leading semiconductor companies in Europe. His background also includes research in computer graphics and VR at the School of Advanced Studies Sant'Anna in Pisa. You can follow him on Twitter @alexvoica.

20 thoughts on “Implementing hybrid ray tracing in a rasterized game engine”

  1. Not sure if anyone is still reading this.. I’m curious about how much space is required to keep a detailed polygon database of every object both in and out of the camera’s view. How does it handle high levels tessellation?

    • A normal game engine keeps seen and unseen objects in memory at all times. So your question is nonsense, unless I misunderstood it.

      • A normal game keeps objects in memory but not the exact location of every single triangle, which is what’s needed to do ray tracing. Ray tracing has much higher memory requirements, that’s a given.

      • No, ray tracing requires much more memory than traditional rasturization because it needs to store detailed triangle information for everything in acceleration structures.
        Here’s an excerpt from a paper about it:
        Most interactive ray tracing approaches have considerable
        memory overhead. This is mainly due to the use of hierarchical
        acceleration structures for efficient ray-intersection
        tests. These hierarchies affect the performance in several
        ways: First, the memory overhead of hierarchical acceleration
        structures can be high in addition to that of scene primitives.
        For example, hierarchies (e.g., kd-trees) optimized for
        run-time performance according to the surface area heuristic
        (SAH) [GS87, MB90, Hav00], tend to have more nodes
        and may take almost as much memory as the triangle primitives.
        Second, hierarchy traversal for ray tracing can have a
        complex memory access pattern. Given the block-based data
        fetching memory hierarchy, the working set size of the ray
        tracer can thus be high. Finally, ray tracers typically require
        random access to the primitives since any triangle can intersect
        with the ray during traversal. Therefore, triangles are
        commonly stored in an indexed triangle list or similar representation,
        which can have large memory overhead in contrast
        to more efficient storage methods used in rasterization.

        • I actually developed a path tracer and a game engine. I’m not just making random claims.
          I’m not arguing whether or not ray tracing uses more memory or not. My point was that normal game engines store all objects in VRAM regardless of their visibility (unless you implement streaming, which is equally applicable to a raytracer).
          “I’m curious about how much space is required to keep a detailed polygon
          database of every object BOTH IN AND OUT OF THE CAMERA’S VIEW.”

          • I’m not a game developer, but I’m fully aware that objects won’t get loaded from hard disk just because you turned your camera in the game. What I meant was that for normal game, you have the G-Buffer which stores detailed information of geometry and materials and it can get quite large in memory, but it’s only for what’s in front of the camera. My understanding is that for a ray tracer, it would need to essentially always have that same information for the entire scene, everywhere, whether it was viewable by the camera or not.

          • It makes no sense to compare G-Buffers and acceleration structures. A ray tracer can use G-Buffers too. And not every game uses a G-Buffer (forward rendering).

  2. Here’s a neat idea that could use ray-tracing to significantly augment a rasterizer:
    Ray Assisted Rasterization
    As scene complexity grows, fragment workload stays constant (ie. TBDR), but the bandwidth and computation cost associated with triangle grows more-or-less proportionally.
    The idea is simple: use the ray-tracer to discover potential intersecting triangles prior to the rasterization step.
    1) In this case, we ray trace over every pixel in a tile, and build a list of candidate triangles according to the ray tracing acceleration structure (eg. BVH).
    2) Once we have traced all pixels, we then take the list of triangles, stream in the appropriate triangle data from memory, transform them and rasterize them with appropriate per-pixel depth info to account for proper draw order.
    3) If there are pixels in the tile that haven’t been rasterized, we can go back to step 1 for those pixels.
    4) Once all pixels in the tile have been rasterized, they are then fragment shaded.
    5) We repeat this for each tile in the target frame.
    This effectively culls triangles based on overdraw. The important point to note is that similar to eliminating hidden-fragment shading calculations, it also eliminates the very need to do any triangle-related work for triangles that are not seen! In other words if the triangle is hidden, it is not even processed. Similarly triangles that are hidden will not read from memory, leading to potentially large bandwidth saving as well.
    The effect on scene complexity and memory bandwidth could be tremendous. Only triangles that will be used in the scene will be streamed from memory into the tracer, and only triangles that will be seen will be calculated. Scene complexity could grow exponentially and be much more memory-capacity bound than bandwidth/performance bound.
    It would also eliminate the need for sorting logic as is the case with the current PowerVR TBDR solution.
    Of course, there are limitations. For one, the scene would have to reside in a structure prior to beginning the render. Additionally, handling of transparency in this scenario would not be straight forward and would have to be solved appropriately (eg. multiple tile buffers). Lastly, for sub-pixel triangle sizes, this could be quite expensive.
    This scheme of Ray Assisted Rasterization has the benefit of saving bandwidth and computation as scene complexity increases.
    I hope that you consider and share this idea with your colleagues, as it is a potentially powerful alternative to the current TBDR scheme of reducing compute complexity and optimizing bandwidth.

      • This is a very exciting prospect. I hope that you are also considering GPUs that scale very far beyond the bounds of mobile GPUs. The next generation of consoles could have Imgtec tracing built-in, and fill-out a 400mm^2 die at 14nm!
        Oh, and coupled with a PoP Hybrid Memory Cube, this GPU could enjoy insane memory bandwidth to augment the insane performance. Low-noise path tracing at interactive framerates would certainly be a realistic proposition with this hardware..
        Let me have my dreams! 🙂

        • Hi,
          The proposed game console would have a multiprocessor, multicluster PowerVR Wizard GPU that would support the latest APIs and scale beyond 1.2 TFLOPS. More about this soon.

        • Not necessarily. There have been several interesting devices that run on Android but use mobile-derived hardware. By upgrading to a multiprocessor, multicluster PowerVR Wizard GPU, they would be able to scale to PlayStation/Xbox levels.

  3. Tracing shadow rays would still need the complete scene as BVH/KD-tree in the video memory. As the g-buffer only contains information about visible objects and not the occluding geometry from light’s perspective. This is one of the limitations of ray-tracing. Although I agree the shadow quality would be very accurate. Modern games already use some kind of ray-tracing for screen space reflections using just the g-buffer.

    • The PowerVR tracer is quite innovative in this regard: It uses a single object that works with OpenGL (similar to a display list, iirc) but that holds the entire scene data (incl. shaders) for the ray tracer. In this way, the rasterizer draws the scene into a GBuffer, and the tracer can then do tracing over each of those framebuffer pixels with complete scene data at hand for better reflections, transparencies, shadows, etc. Submission of objects to this unified data structure makes the marriage of rasterizing and ray tracing quite straight forward.*
      Tracing should also save a lot of bandwidth, not requiring a scene to be rendered from multiple perspectives.
      >Modern games already use some kind of ray-tracing for screen space reflections using just the g-buffer
      An important thing to note is this: this is a mobile platform, and despite relatively constrained bandwidth and compute, the on paper tracing performance is nothing short of amazing. While modern desktop/console games may use tracing to get an approximation of light ray intersection from some buffer, cube/sphere-map, etc, it is at best a hack an in no way comparable to the flexibility, accuracy, and simplicity of real ray tracing.
      * corrections appreciated where applicable

      • Cool. Good to know. It seems like OpenGL and OpenRL work together pretty well in this regard.
        I agree the SS reflections are hacky and this architecture would provide a more simplified way to perform accurate ray-tracing. I would like to see some accurate voxel cone tracing for diffuse global illumination using this.

        • Me too. I’m sure that a great many things can be explored with this architecture, and who knows? I remember reading that this can handle ~5 rays per pixel, future generations at smaller process-nodes would be capable of much more (including tablet specific SoCs with different power constraints), and we may have something that is directly capable of path tracing.

    • Screen space reflections is a very fast way to get reflections going, but it is also quite hacky from an imaging point of view. That’s OK for games since people aren’t there to check if reflections are accurate or not as long as the scene is believable. Shadows need something more orthogonal and accurate.
      That said, if you take into account the light fall-off distance, and take into account that not all lights require soft shadows, it is possible to do RT shadows without having the whole scene on chip, although one needs more geometry than what’s visible on screen.
      Global illumination would also benefit from the ability to sample based on close proximity geometry. it would be much more straightforward than using other methods such as cone-tracing or light propagation volumes. Both are proven to be effective, but generate their fair share of corner-cases etc.
      I’m pretty excited by what Imgtec has done, although we’ll have to see if RT gets adopted by NV and AMD.


Please leave a comment below

Comment policy: We love comments and appreciate the time that readers spend to share ideas and give feedback. However, all comments are manually moderated and those deemed to be spam or solely promotional will be deleted. We respect your privacy and will not publish your personal details.

Blog Contact

If you have any enquiries regarding any of our blog posts, please contact:

United Kingdom

[email protected]
Tel: +44 (0)1923 260 511

Search by Tag

Search by Author

Related blog articles

b series imgic technology hero banner

PowerVR SDK and Tools 21.1 Out Now – including ray tracing simulator

The sun is starting to come out, people are returning to society, and potentially most exciting; we at DevTech are releasing a new version of the PowerVR SDK and Tools. The PowerVR SDK contains not only our graphics development framework, but also a handful of example applications that show you not only how to implement certain techniques, but also make the most out of your PowerVR platforms. Between the modules that help you deploy cross-platform or libraries that let you focus on the actual code rather than boilerplate setup; our SDK has everything you need to hit the ground running with making a high-performance application. We’re starting with the SDK, rather than the tools this time, because this first talking point is a pretty big one, we think. 

Read More »
Apple M1 image

Why you should be running your iOS apps on your new Apple M1 laptop

Towards the end of last year, Apple released the latest version of its Apple MacBook Pro and Macbook Air laptops. This release was notable as with these brand-new laptops Apple made a significant change – the processor inside was based on its own M1 chip rather than the Intel architecture that it had been using since 2006. Since its release, the Apple M1 has been widely hailed for its performance, outstripping Intel in all the major benchmarks and all in a cool, quiet package with low power consumption.

Read More »
ray tracing Detail Lighting

Imagination collects 5G Cloud Gaming Industry Alliance award for ray tracing

Imagination pioneered hardware-based ray tracing technology several years ago with a focus on an ideal blend of performance, quality power efficiency and the IP is available for licensing. Its efforts have been recognised by the 5G Cloud Gaming Industry Alliance (5GCGA), a new cross-industry collective that aims to develop the 5G cloud game industry worldwide.

Read More »


Sign up to receive the latest news and product updates from Imagination straight to your inbox.