Develop · Build
How it came together.
- 01
Start where the engine already is
The first pass used the renderer's own relighting: derive each Gaussian's normal in the shader from its rotation and scale, then shade the captured colour against a sun vector from a real solar ephemeris. It works, it runs in real time on millions of Gaussians, and it is the thing shipped on this page. What it cannot do is subtract, which is what sets up everything below.
3D & Spatial - 02
Try to recover albedo properly, and stop after two runs
Inverse rendering is the principled answer: solve for geometry, material and lighting together. Two retrains ended at test PSNR 14.7 and then 14.4 with a denser 300,000-point initialisation and segmentation masks, with the recovered environment map collapsed to flat white both times. The cause is not the implementation. A scene observed under exactly one lighting condition does not constrain the problem, so a third run was logged as do-not-retry rather than attempted.
Research - 03
Ask the delighter to agree with itself
A learned delighter separates one photograph into albedo and shading convincingly. The test that matters is whether it gives the same albedo for the same surface seen from a different camera, because a splat is built from many views. Median cross-view albedo standard deviation came back at 0.223, with 2.2 percent of Gaussians consistent below 0.10. That is a fail, and it is a fail found by measuring rather than by looking.
Research - 04
Keep the one frame that worked
The same pass that failed the gate produced the single genuine result in the corpus: a frame with the baked cast shadow actually removed, at 0.3 seconds a frame. It is on this page as the before and after. One frame is not a pipeline, and it is reported as one frame, but it is the evidence that the ceiling is a data problem rather than an impossible one.
3D & Spatial - 05
Get the normals right in the browser, not in a sidecar
The web viewer reorders splats as it loads, so a normal computed against the file's ordering lands on the wrong Gaussian and the scene turns to speckle. That kept per-Gaussian relighting off this site for months. A Gaussian's covariance is its rotation and scale already combined, so the eigenvector of its smallest eigenvalue is the surface normal, and the covariance array is in the renderer's order by construction. No extra download, and no alignment left to get wrong.
Frontend Engineering - 06
Find out what the real limit is before spending more on it
The obvious suspect for reconstruction quality was camera pose precision. A native-pose ablation overturned it: the limit is how densely the scene was observed, not how accurately the observations were registered. That moves the next experiment out of the renderer and back to the scanner, which is the more expensive answer and the correct one.
Research
Develop · Forks
Decisions on the record.
The few calls worth defending. Each one is a fork; the other branch would have been a different project.
Decision · 01
Normals from covariance, not from a sidecar file
The web viewer reorders splats as it loads, so a normal computed against the file's own ordering lands on the wrong Gaussian and the scene turns to speckle. That had the feature blocked for months. A Gaussian's covariance is its rotation and scale already multiplied together, so the eigenvector of its smallest eigenvalue is the surface normal, and the covariance array is in the renderer's order by construction. No sidecar, no alignment to get wrong.
Decision · 02
Abandoning inverse rendering after two runs, not ten
Recovering geometry, material and lighting from a scene captured under exactly one lighting condition is under-determined. Two retrains said so in the same voice: test PSNR 14.7, then 14.4 with a denser initialisation and segmentation masks, environment map flat white both times. The third run would have failed for the same reason, so it was logged as do-not-retry instead of attempted.
Decision · 03
Shipping cast shadows off
A cast shadow computed at render time adds to the shadow already baked into the capture, so the wall gets two. Two ways of computing one were tried and both failed, from opposite directions: reading occlusion off a proxy mesh put the mesh's own lumps on the wall under raking light, and reading it off per-Gaussian normals put grain there instead. Both infer shadow from the wrong thing, a stand-in surface or a noisy quantity. PlayCanvas has since shipped the proxy-mesh route as a documented engine feature, and its own guidance names the same limit in the same terms: where the proxy's silhouette diverges from the splat surface, the lighting boundaries land in the wrong place. The fix is to compute visibility from the splat itself, by rendering it from the light's viewpoint, and it is specified and not yet built. Until it is, the default is a movable light with cast shadow off, which is a mood system rather than a lighting simulation, and it is described that way rather than oversold.
Decision · 04
Diffuse only, rather than modelling specular badly
Specular response needs a view-dependent term the capture cannot supply reliably, and a wrong highlight reads as a defect where a missing one reads as soft light. Half-Lambert with an ambient floor keeps surfaces facing away from the sun grey rather than black, which is what an interior actually looks like and what the data can support.
Decision · 05
Capture is the ceiling, and an ablation proved it
The obvious suspect for reconstruction quality was camera pose precision. A native-pose ablation overturned that: the limit is how densely the scene was observed, not how accurately the observations were registered. That moves the next experiment from the renderer to the scanner, which is a more expensive answer and the correct one.