Loading
Research · Thread 04
Relighting baked light
A scan records the light that was there. Changing it later is a different problem from rendering it.
A Gaussian splat stores what the cameras saw, which is radiance: colour with the room's own light already inside it. Relighting wants albedo, the colour a surface would be under any light at all. Multiplying radiance by a new sun therefore lights the scene twice, and the shadow that was on the wall at capture time stays on the wall at every hour you dial up. Four method families were tested against that: engine-side per-Gaussian shading, inverse-rendering retrains, learned delighting, and real-time cast shadows. One shipped, one produced a single genuine result, and two were logged as do-not-retry with the numbers that closed them.
Paper to shipped
Reference
Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis (Ke et al., 2025)The delighter used here, in its IID-Lighting form: it splits a photograph into albedo, diffuse shading and a non-diffuse residual. Per-image it is convincing. Across the views of one scan it is not, which is the finding below.
View abstract
Marigold adapts pretrained latent diffusion image generators to dense image-analysis tasks with a small synthetic training set and modest compute, changing as little of the generative prior as possible. The family covers monocular depth, surface normals and intrinsic image decomposition; the IID variants recover albedo together with either BRDF material properties or diffuse shading and a non-diffuse residual. arXiv:2505.09358.
Reference
IRGS: Inter-Reflective Gaussian Splatting with 2D Gaussian Ray Tracing (Gu, Wei, Zeng, Yao, Zhang, CVPR 2025)The inverse-rendering reference point, and the one that was actually retrained here. It ray-traces 2D Gaussians to evaluate the rendering equation without simplifying visibility, which is the right shape of answer if the capture can support it. The same group published IRGS++ on 24 July 2026, a week after these retrains stopped. It widens the method to glossy and metallic materials and cuts the cost of relighting under new illumination. What it does not touch is the constraint that closed this line, which is how the scene was lit when it was scanned.
View abstract
Prior 3DGS inverse-rendering methods either simplify the rendering equation or approximate incident light with learnable parameters, giving inaccurate material and lighting estimates. IRGS applies the full rendering equation and computes incident radiance on the fly using differentiable 2D Gaussian ray tracing, whose well-defined ray-splat intersections allow tracing directly on a pretrained 2D Gaussian splatting model with minimal quality loss. arXiv:2412.15867.
Benchmark
Two inverse-rendering retrains, both abandoned on the numbersTest PSNR 14.7, then 14.4 on a second run with a denser 300,000-point initialisation and segmentation masks. The recovered environment map collapsed to flat white both times. The cause is not the implementation: recovering geometry, material and lighting from a capture lit exactly one way is under-determined, and no amount of retraining adds the missing observation.
Benchmark
A kill gate for learned delighting, and the one frame that passedDelighting every view independently and asking whether the views agree: median cross-view albedo standard deviation 0.223, with only 2.2 percent of Gaussians consistent below 0.10. That is a fail. It is also how the same delighter produced the single real breakthrough in the corpus, a frame with the baked cast shadow genuinely gone, at 0.3 seconds a frame.
Shipped
Per-Gaussian relighting in the browser, diffuse only, shadows offWhat survived: each Gaussian shaded against a real solar position from its own surface normal, derived from its covariance so it stays correct after the renderer reorders the cloud. Two real-time cast-shadow approaches failed from opposite directions, a proxy mesh too coarse and per-Gaussian normals too noisy, so the shipped default is a movable light with cast shadow off. It is a mood system that is honest about being one.
Reference
LightBridge: Feed-Forward Generative Relighting for 3D Gaussian Splatting (Cao, Cheng, Du, Li, Cai, Liu, 2026)Where the field went while this sat. Published 2 September 2026, it relights a whole splat in one forward pass, with no per-scene optimisation, by learning the transport from source lighting to target lighting and propagating it across the Gaussians. That sidesteps the exact failure logged above: the two retrains here failed because per-scene inverse rendering cannot recover lighting it never observed, and a model trained across many illuminations brings that missing observation with it instead of trying to infer it. It is the honest successor to the abandoned path, and it is untested on this data.
Proposed
The next move is capture, not codeA native-pose ablation overturned the pose-precision hypothesis: the limit is view density at capture, not registration error. Re-scanning under flat light, at blue hour or with sheers drawn, removes more baked shadow than any renderer change on the existing data can. Standard relighting benchmarks come next so these numbers can be compared with published work instead of only with each other.