Inverse Rendering in Practice

With correct and affordable gradients, the applications follow from one capability: fitting physical scene parameters to observed images.

Material capture. Photograph a real surface under known lighting and recover its BSDF. The output is a physically meaningful material description, usable in any renderer and under any new lighting, which is what distinguishes it from copying pixels.

Geometry reconstruction. Recover shape from images, with the advantage that the model accounts for shading, shadows, and interreflection rather than treating them as nuisances.

Lighting estimation. Recover the illumination of a scene from photographs, which is what lets a virtual object be inserted into a real photograph and lit consistently.

Computational design. Run the optimisation in the other direction: rather than fitting to a photograph, specify a desired appearance and solve for the physical configuration that produces it. Caustic design, where a transparent surface is shaped so refracted light forms a chosen image, is the striking example.

That last category is worth noticing, because it uses the same machinery to design something manufacturable rather than to reconstruct something observed.

Neural radiance fields and Gaussian splatting reconstruct 3D scenes from photographs and are far more widely used than physically based inverse rendering. It is worth being precise about how they differ, because they are often conflated.

Both families optimise a scene representation by differentiable image formation. The difference is what is being represented and what the rendering model is.

NeRF-style methods represent a scene as a field of density and view-dependent colour, rendered by volumetric integration along rays. Gaussian splatting represents it as a cloud of Gaussians rasterised efficiently.

Neither separates material from illumination. A NeRF learns that a surface looks a particular way from a particular angle under the lighting present when it was captured. It does not know whether that appearance came from a shiny surface or a bright light, because it never models light transport.

The consequence is practical. Relight a NeRF and you cannot, at least not without additional machinery, because the information required was never disentangled. Physically based inverse rendering recovers separated, physical parameters, so relighting, material editing, and transfer to other scenes all work by construction.

The two are converging, with work using NeRF representations as environment emitters inside physics-based inverse rendering pipelines.

The deepest difficulty is not computational. Many different scenes produce the same image.

The canonical ambiguity is between material and light. A bright surface under dim light and a dim surface under bright light can be pixel-for-pixel identical. Nothing in the image distinguishes them, so no optimiser can either.

Other ambiguities are equally fundamental. Shape and normals trade off, since a smooth surface with detailed normals can mimic detailed geometry. Roughness and light size trade off, since a blurry highlight may come from a rough material or a large light source.

This is not a defect of the method. It is a property of the data, and recognising that changes what you do about it.

The available responses all amount to adding information. Capture more views or vary the lighting between shots, so different configurations become distinguishable. Constrain with priors, penalising implausible materials or encouraging smooth geometry. Fix what you know, since calibrating the lighting in a capture rig removes the largest ambiguity outright.

The practical rule: if your reconstruction produces implausible material and lighting values that nonetheless render correctly, the problem is under-constrained rather than under-optimised.

The loss landscape of inverse rendering is non-convex, and its non-convexity has a specific and awkward character.

Start with a sphere and try to optimise it into a chair. Gradient descent moves vertices continuously, so the mesh must deform through every intermediate shape. If a low-loss barrier lies between them, it gets stuck partway, producing the melted, blobby results familiar to anyone who has tried this.

Worse, topology cannot change by gradient descent. A sphere has no holes. A chair does. No continuous deformation of vertex positions creates one, and no gradient will suggest it, so a mesh-based optimisation is confined to its initial topology permanently.

The standard mitigations are worth knowing because they generalise.

Coarse to fine. Optimise a low-resolution mesh first, then subdivide. Early optimisation sees a smoother landscape with fewer local minima.

Better initialisation. Start from a rough reconstruction obtained another way rather than a primitive.

Alternative representations. Optimise a signed distance field or a volumetric density, which can change topology freely, then extract a mesh at the end. This is one reason volumetric representations became so popular.

Regularise the mesh, since unconstrained vertex optimisation produces tangled self-intersecting geometry.

The four failure modes have different signatures and completely different fixes, so identifying which one you have is most of the work.

The fourth branch of that diagram deserves its own treatment, because it is the failure people diagnose last and should check first.

Inverse rendering assumes your forward model can actually produce the target image. Where the real world contains physics your renderer omits, the optimiser will compensate by distorting whatever parameters it does have.

The common omissions are specific. Subsurface scattering in skin, marble, and leaves, where light enters and re-emerges elsewhere. Participating media such as haze. Spectral effects including dispersion and fluorescence, if you render in three colour channels. Polarisation. And camera effects: lens flare, chromatic aberration, sensor noise, and the tone mapping applied before you ever saw the image.

The failure mode is insidious because the optimiser succeeds at what you asked. Told to match pixels with a model lacking subsurface scattering, it will find a strange BSDF and an implausible light configuration that together reproduce the appearance. Low loss, wrong physics, useless parameters.

The practical discipline is to validate on synthetic data first: render a known scene with your own renderer, then try to recover its parameters. If that fails, the problem is in your pipeline. If it succeeds and real data fails, the gap is model mismatch.

What a robust inverse rendering setup actually looks like, assembled from the failure modes above.

1. Validate on synthetic data
   Render a known scene, recover its parameters, confirm it works.

2. Check geometry gradients are non-zero
   The single most common silent failure.

3. Fix everything you know
   Calibrated camera, measured lighting, known scale.
   Each fixed parameter removes an ambiguity.

4. Optimise in stages
   Lighting and coarse materials first; geometry later;
   fine detail last. Do not release all parameters at once.

5. Coarse to fine on geometry
   Low-resolution mesh, then subdivide.

6. Regularise
   Smoothness on geometry, plausible ranges on materials.

7. Ramp sample counts
   Cheap noisy gradients early, precise ones near convergence.

Step 4 carries more weight than its position suggests. Releasing every parameter simultaneously lets the optimiser find compensating errors: geometry absorbing a lighting mistake, materials absorbing a geometry mistake. Staging the optimisation constrains it toward the physically sensible solution among the many that fit.

Step 3 is where the ill-posedness is genuinely defeated, and it is usually cheaper than any algorithmic improvement.

Four claims, and what the whole approach is for.

Rendering is a high-dimensional integral estimated by Monte Carlo. The rendering equation has no closed form, sampling is the only viable approach, and every image is a noisy but unbiased estimate.

Differentiating it is not routine automatic differentiation. Visibility discontinuities put a boundary term on a set of measure zero that random sampling can never hit, so geometry gradients come back silently zero unless edge sampling, reparameterization, or warped-area methods supply the missing term.

Efficiency required rethinking the backward pass. Radiative backpropagation treats it as an adjoint transport simulation rather than a recorded tape, path replay reduces memory to constant by storing seeds instead of paths, and a purpose-built JIT compiler addresses GPU thread divergence.

The remaining difficulties are about the problem, not the gradients. Inverse rendering is ill-posed, has a landscape with topology traps, and fails quietly when the forward model omits physics the real scene contains.

The reason to accept all of that is the one thing this approach gives that image-space methods do not: physically separated parameters. Recover a material and it is a material, valid under lighting you have not captured and in scenes you have not built.