03·3D & Spatial
Relighting a Real Scan
A Gaussian-splat capture arrives with its lighting baked in, frozen at whatever time of day the scanner ran. This is a month of trying to unfreeze it: per-Gaussian relighting driven by a real solar ephemeris, so the sun crossing the scene moves the way the sun actually moves, plus three other approaches that failed and the numbers that closed them. You can drag the sun across the live scan below. What you cannot do is remove the shadow that was already on the wall, and most of the work here is about why.
ResultThe loop holds end to end: a real capture, a real sun, real-time relighting, in an ordinary browser with no install.
[ Walk the live 3D scan ]01Discover
Why it exists.
A Gaussian splat is built from photographs, so it inherits their light. The scan holds radiance: the colour a surface had under the lamps and sky that happened to be there. Relighting wants the other thing, albedo, the colour a surface would be under any light at all. Nothing in the capture separates the two, and every method tried here ran into that in a different way.
The practical consequence is easy to state and hard to get around. Multiply the captured colour by a new sun and the room is lit twice: once by the light that was there and once by the light you just added. The shadow that fell across the wall at capture time stays on that wall at every hour you dial up, because it is not a shadow any more, it is paint.
The reason to care is commercial rather than academic. A scan of a residence sold at a fixed hour is a scan that shows badly at every other hour, and the fix a client asks for first is the sun.
See it live
Per-Gaussian relighting, live
The same capture as the viewer case study, relit rather than colour-graded: 2,049,226 Gaussians on desktop, 511,668 on a phone. Each one is shaded against a real solar position using a normal derived from its own covariance, so vertical surfaces respond to the sun's azimuth and not just its height. Drag the sun slider and watch which wall brightens. WASD to walk, drag to look.
02Define
The brief.
Take a 3D scan whose lighting is frozen at the hour it was captured, and let someone move the sun across it in a browser without the room falling apart.
03Develop
What it took
Skills behind it.
Primary discipline plus the support stack.
Skills demonstrated
- 3D & SpatialPrimary
Relit millions of Gaussians individually from the scan's own geometry, rather than lighting a proxy mesh and hoping the splats followed.
- ResearchPrimary
Framed the question so it could come back negative, and it did: three of four method families failed, each closed by a number rather than by a hunch, and each logged as do-not-retry with the reason.
- Frontend EngineeringSecondary
Solved the ordering problem that kept per-Gaussian normals out of the web viewer, so the demo on this page is the technique and not a mock-up of it.
- Data PipelinesSupporting
Decoded the proprietary capture format from its published spec, since nothing off the shelf would read it into the engine.
Develop · Notes
The build, in full.
How it came together and the decisions on the record - kept off this page so the case study stays a read, not a scroll.
Read the full build notesDevelop · Compare
Drag to compare.
Same scan, two states. Move the divider with the mouse, touch, or arrow keys.
CapturedDelit
04Deliver
What shipped.
The loop holds end to end: a real capture, a real sun, real-time relighting, in an ordinary browser with no install. The limits are real too. A scan records radiance, not albedo, so the light that was in the room at capture time is already in the data and relighting adds to it rather than replacing it. Two inverse-rendering retrains ended at test PSNR 14.7 and 14.4 with the environment map collapsed to flat white. Learned delighting failed a cross-view consistency gate at a median albedo standard deviation of 0.223, and produced, in the same pass, the single frame in the whole corpus with the baked cast shadow genuinely removed. Two real-time cast-shadow approaches failed from opposite directions, one reading occlusion off a proxy mesh too coarse to carry it and one off per-Gaussian normals too noisy to trust, so the shipped default is a movable light with cast shadow off. PlayCanvas has since shipped that same proxy-mesh route as a documented engine feature and names the same limit in its own guidance, which is the useful kind of confirmation: the constraint is in the technique, not in this implementation of it.
What ships is a mood system that is honest about being one: a real sun, a real per-Gaussian response, and no claim to have removed light that is already in the data. The negative results are the more useful half. Two abandoned retrains, a delighting gate failed at 0.223, and two cast-shadow approaches that failed from opposite directions together say something a working demo cannot, which is where the boundary actually sits and what it would take to move it. The next move is a recapture under flat light and a run against the standard relighting benchmarks, so these numbers can be compared with published work instead of only with each other. The method to test against is no longer the one abandoned here: relighting has since moved feed-forward, learning the change of lighting across many scenes rather than solving for it inside one, which is a direct answer to why the retrains failed.
By the numbers
Largest capture
the outdoor site, in one continuous scan; the demo above is a smaller interior
Light source
solar ephemeris, so azimuth and altitude both move
Delighting kill gate
median cross-view albedo deviation; 2.2 percent passed
Research origin
The paper-to-product thread this project sits inside. The full chain, paper through benchmark to shipped, is on the thread.
- Relighting baked lightSee thread
A scan records the light that was there. Changing it later is a different problem from rendering it.
- 3D Gaussian SplattingSee thread
Turning ordinary photographs of a real place into a 3D scene you can walk through in a browser.
Frames
Selected from the work.
3 frames
01Cool. The same geometry, the same baked shadows, a different sun. 02Midday. Note the doorway: the light through it never moves, because it is in the capture. 03Golden. Convincing at a glance, and the reason the failures needed measuring rather than eyeballing.
See also
3D & Spatial
3DGS Research Pipeline
A reproducible harness with 26 Gaussian-splat engines wired into it, scoring which reconstruction survives a real luxury interior.
3D & Spatial
Barnes Vantage
An international portal for viewing real Dubai residences in 3D: walk a real scan yourself, or have a broker drive the same walkthrough live.