Programme note, index only / Immersive 3D / Research

Relighting a Real Scan

Per-Gaussian relighting of a real 3D Gaussian-splat capture in the browser, driven by a real solar ephemeris, with the negative results reported alongside.

My role
Investigation lead: built the per-Gaussian relight path, wired the ephemeris, ran the inverse-rendering and delighting experiments, and wrote up what held and what did not.
Made at
BARNES Dubai
Built
2026
Status
Archived
Access
The demo on this page is public; the client capture it was developed on is private
Stack
Gaussian splatting, WebGL, PlayCanvas, GPU shaders, Solar ephemeris, Marigold-IID, PyTorch
A 3D Gaussian-splat capture of an apartment interior, relit warm: a dining table, a sofa, and a marble kitchen counter in the foreground.

Chapter 01 of 04

The brief

A scanned home shows one light only: the light of the hour it was scanned.

A buyer cannot see the living room at sunset. This is a month of trying to change that: per-Gaussian relighting driven by a real solar ephemeris, so the sun crossing the scene moves the way the sun actually moves, plus three other approaches that failed and the numbers that closed them. You can drag the sun across the live scan below. What you cannot do is remove the shadow that was already on the wall, and most of the work here is about why.

A Gaussian splat is built from photographs, so it inherits their light. The scan holds radiance: the colour a surface had under the lamps and sky that happened to be there. Relighting wants the other thing, albedo, the colour a surface would be under any light at all. Nothing in the capture separates the two, and every method tried here ran into that in a different way.

The practical consequence is easy to state and hard to get around. Multiply the captured colour by a new sun and the room is lit twice: once by the light that was there and once by the light you just added. The shadow that fell across the wall at capture time stays on that wall at every hour you dial up, because it is not a shadow any more, it is paint.

The reason to care is commercial rather than academic. A scan of a residence sold at a fixed hour is a scan that shows badly at every other hour, and the fix a client asks for first is the sun.

A real 3D Gaussian-splat interior capture with a sun slider, relightable live in the browser.
The same capture as the viewer case study, relit rather than colour-graded: 2,049,226 Gaussians on desktop, 511,668 on a phone. Each one is shaded against a real solar position using a normal derived from its own covariance, so vertical surfaces respond to the sun's azimuth and not just its height. Drag the sun slider and watch which wall brightens. WASD to walk, drag to look.

Chapter 02 of 04

The problem

Take a 3D scan whose lighting is frozen at the hour it was captured, and let someone move the sun across it in a browser without the room falling apart.

Chapter 03 of 04

The build

3D & Spatial
Relit millions of Gaussians individually from the scan's own geometry, rather than lighting a proxy mesh and hoping the splats followed.
Research
Framed the question so it could come back negative, and it did: three of four method families failed, each closed by a number rather than by a hunch, and each logged as do-not-retry with the reason.
Frontend Engineering
Solved the ordering problem that kept per-Gaussian normals out of the web viewer, so the demo on this page is the technique and not a mock-up of it.
Data Pipelines
Decoded the proprietary capture format from its published spec, since nothing off the shelf would read it into the engine.

Same scan, two states. Move the divider with the mouse, touch, or arrow keys.

A scanned interior wall with a sculpture on a shelf, throwing a hard cast shadow up and to the left, the wall around it graded warm to cool.CapturedDelit
After: The same wall after delighting: the cast shadow is gone and the wall reads as one flat cream surface.
Learned delighting: the one frame that worked

Build notes. How it came together and the decisions on the record.

[ Read the full build notes ]

Chapter 04 of 04

What it did

The loop holds end to end: a real capture, a real sun, real-time relighting, in an ordinary browser with no install.

The limits are real too. A scan records radiance, not albedo, so the light that was in the room at capture time is already in the data and relighting adds to it rather than replacing it. Inverse rendering was abandoned after two retrains (the scores are under Decisions). Learned delighting failed a cross-view consistency gate at a median albedo standard deviation of 0.223, and produced, in the same pass, the single frame in the whole corpus with the baked cast shadow genuinely removed. Two real-time cast-shadow approaches failed from opposite directions, one reading occlusion off a proxy mesh too coarse to carry it and one off per-Gaussian normals too noisy to trust, so the shipped default is a movable light with cast shadow off. PlayCanvas has since shipped that same proxy-mesh route as a documented engine feature and names the same limit in its own guidance, which is the useful kind of confirmation: the constraint is in the technique, not in this implementation of it.

What ships is a mood system that is honest about being one: a real sun, a real per-Gaussian response, and no claim to have removed light that is already in the data. The negative results are the more useful half. Two abandoned retrains, a delighting gate failed at 0.223, and two cast-shadow approaches that failed from opposite directions together say something a working demo cannot, which is where the boundary actually sits and what it would take to move it. The next move is a recapture under flat light and a run against the standard relighting benchmarks, so these numbers can be compared with published work instead of only with each other. The method to test against is no longer the one abandoned here: relighting has since moved feed-forward, learning the change of lighting across many scenes rather than solving for it inside one, which is a direct answer to why the retrains failed.

17.7M
Largest capturethe outdoor site, in one continuous scan; the demo above is a smaller interior
Real sun
Light sourcesolar ephemeris, so azimuth and altitude both move
  1. 01Cool. The same geometry, the same baked shadows, a different sun.
  2. 02Midday. Note the doorway: the light through it never moves, because it is in the capture.
  3. 03Golden. Convincing at a glance, and the reason the failures needed measuring rather than eyeballing.

Research origin.

[ Relighting baked light ]
A scan records the light that was there. Changing it later is a different problem from rendering it.

[ 3D Gaussian Splatting ]
Turning ordinary photographs of a real place into a 3D scene you can walk through in a browser.

Next on the programmeImmersive 3D / Research / 2025 to 2026
A detail of a vector floor plan produced by the scan-to-floor-plan pipeline: two bedrooms side by side with their areas in square metres and square feet, wall thicknesses, door swings, a dimension chain along the top, a north arrow, and the upper edge of the master bedroom below.

Scan to Floor Plan. Turning a 3D scan into a drawn-to-scale floor plan, and the measured finding that the machine-learning version lost to a far simpler method.

[ Read the programme note ]

Contact

Pavlo Puzikov, a black and white studio portrait in a pinstripe suit.

Pavlo Puzikov

Marketing and innovation: immersive 3D, applied AI, generative content

Write to

pavlopuzikov@gmail.com

Open to
Freelance projects, remote. Full-time roles, onsite or hybrid in the EU.
Based in
Bulgaria
Most recent role
Marketing and Innovation Lead, BARNES Dubai, 2026. Product Innovation Researcher there from 2025.
Education
BCom, Marketing and Management, University of Auckland, 2020 to 2023
Languages
English, Russian