I’ve just learnt what Gaussian Splatting is, and I fear it may become my entire personality. At the risk of oversimplification, Gaussian Splatting is a rendering technique that gives you ‘photo-real’ graphics at a fraction of the resource cost required by a number of other mainstream game development options. That fact alone is enough to encourage me to don the mantle of ‘Gaussian girlie’—at least for this weekend column—and take a deep dive into the subject with an expert.
So, first things first, it’s pronounced ‘GOW-see-an,’ and not ‘GAWE-shun’ as I have been saying for the last week. This is because it’s derived from the name of the German mathematician that discovered them, Johann Carl Friedrich Gauss. Don’t worry, I’m going to keep the maths to a minimum here—the passing GCSE grade I got over a decade ago has seen to that. But allow me to recap what sparked my splatting obsession about a week ago: specifically, this photorealistic FPS that runs in-browser thanks to ‘Gaussian Splatting’.
The game logic of that straightforward shooter was put together by Iakov Sumygin, a Software Engineer at Snap Inc. Sumygin was kind enough to put me in touch with the artist behind the browser game’s incredible environment, Christoph Schindelar. Schindelar is a self-described ‘Scan-Specialist’ who has previously worked at Quixel, the Epic-owned company that touts it has the largest photogrammetry asset library in the world.
“For a layperson, 3D Gaussian Splatting can be compared to an extremely advanced sprite-based particle system,” Schindelar tells me, “In fact, some early Unreal Engine implementations rendered Gaussian Splats through Niagara, Unreal’s internal particle system.”
A Gaussian in this context does not behave much like the typical sort of polygons you’d see on, say, a 3D model of a tree. As I am an English graduate, I’m going to deploy a simile: you can think of individual Gaussian Splats as not unlike dandelion tufts caught on the breeze, with a more opaque central point and a fuzzier outer edge. Now, for a metaphor: One dandelion seed isn’t much on its own, but a soft shape comes into view as you collect many of them together.
(Image credit: Obsidian Entertainment)
If you look back at the aforementioned browser-based FPS, you’ll notice some parts of the environment have an almost pointillism look or impressionistic fuzz to them, hinting at all the 3D Gaussians coming together to create the full, high-resolution, real-time asset.
Now, I did say I’d keep the maths chat to a minimum, but it is necessary if you want a bit more of a technical understanding. Schindelar continues, “Each splat is a 3D Gaussian that projects to an elliptical footprint on screen. It can be stretched differently in different directions, has opacity, and can change its color depending on the viewing angle using spherical harmonics.”
So yes, that’s a bit more complicated than a dandelion seed on the breeze. The secret, resource-saving sauce is this, though: “Since rendering mostly means projecting and blending these splats on the GPU, it can be extremely fast.”
Gaussian Splatting isn’t just speedy for your rig either; as Schindler argues, it can also be a game changer for a production pipeline.
(Image credit: Iakov Sumygin)
Schindelar has worked with various scanning techniques over the course of his career, “creating game-ready scanned assets and scan-based displacement textures.” With more traditional scanning and rendering methods, this process involves “a lot of manual and semi-automatic work: optimization, cleanup, retopology, UVs, texture projection, baking, material setup, and many other steps before the asset is actually usable in production.”
You can see examples of two different scanning approaches in Schindelar’s portfolio. This RAW model of a child’s hiking boot used “laser-scanning for a clean mesh, photogrammetry for sharp and color-corrected textures, photometric stereo for detail normals” and a polarization-based scan to reduce unwanted glare. Due to the nature of the scanning techniques used, the laces were “separated and added later to the model.” Handily, Schindelar also created a splat-based model of the shoe, too, so you can compare and contrast. The key difference, obviously, is that you wouldn’t be able to turn over the RAW model in a browser window like you can with the splat.
To be clear, Schindelar acknowledges that “Optimized traditional mesh-and-texture workflows can also be extremely fast.” However, a pipeline based around Gaussian Splatting can streamline a lot of that aforementioned manual and semi-automatic work.
Schindelar goes on to elaborate, “In a Gaussian Splatting pipeline, many of these steps can be almost completely bypassed. Instead of rebuilding the captured reality into clean geometry and textures, the training process directly turns the captured image data into a real-time representation. That means a huge amount of traditional scan-processing work can be avoided—which is one of the biggest practical advantages of [Gaussian Splatting].”
(Image credit: Iakov Sumygin)
Open source engine PlayCanvas is what Iakov Sumygin used to build that browser-based FPS. Resources like this strengthen Schindelar’s case, particularly as the engine just introduced SplatTransform 2.0, a tool that offers “fully automated, lightning-fast generation of high-quality collision for your splats.” Without a collision mesh, players could otherwise phase through the environment, so this is yet another option that streamlines the pipeline between scan and interactive assets.
“Gaussian Splatting training—meaning the reconstruction process after capture—can reproduce real-world appearance in ways that traditional scanning methods struggle with or cannot handle properly,” He tells me, “We can now capture and represent things like hair, semi-transparency, translucency, subsurface scattering, fine foliage, and other complex visual phenomena that are extremely difficult to reconstruct as clean geometry with traditional texture workflows.”
“This direct connection between captured real-world data and a production-ready, real-time representation is what makes Gaussian Splatting so interesting,” Schindelar says, “It is not just a rendering trick—it changes the entire capture-to-delivery pipeline.”
gaussiansplatting from r/GaussianSplatting/comments/1t4f4xr/splattransform_20_automated_collision_generation
As a fan of FMV games, the potential applications of Gaussian splatting for video and game development are extremely exciting to me. I love FMV games because live action footage can be a cost-effective way to create visuals for a story-heavy game, and splats could give the whole genre a new lease on life. While something like photogrammetry requires a still subject and an expensive rig for accurate capture, Gaussian splatting is much more forgiving. In theory, multiple passes of a building captured with a drone-based camera or even a short walk filmed with your smartphone could be used to create a Gaussian splat.
In these ways, Gaussian splatting could be invigorating for small teams and game projects. However, ‘photo-real’ environments are not always the most readable for a player—hence the use of the much derided yellow paint to highlight interactables or otherwise indicate paths forward in modern games that attempt a photorealistic artstyle. That said, there’s not yet been a major release that has used Gaussian splatting to create its 3D assets. Time can only tell whether the tech will escape its techy niche and wow a wider audience. In the meantime, there’s always SuperSplat and the Gaussian splatting subreddit for your viewing pleasure.
